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EDITORIAL 


Towards the end of 1938 J. B. Rhine asked us to take over the 
Journal of Parapsychology. He wished to have time for his own re- 4 
| search free from the interruptions attendant upon editing a journal. 
#i\ And in view of Dr. McDougall’s death, he felt that the burden of the 
/ journal, if he continued alone, would be unbearable. We decided to 

accede to his request provided certain changes in the journal plan could 
be introduced. The most important of these changes are as follows: 
ny (1) The journal is to appear only twice a year, each number containing 
| about 150 pages, including a long experimental monograph and a num- 
ber of short papers. (2) Emphasis is to be upon consistent technical 
reporting with very detailed accounts of experimental and statistical 
method, but the general reader’s needs will be met by prefacing each mon- 
“4 ographic report by a brief abstract. 
57 We should have hesitated to undertake so large a task were it not 
wh for the generous assistance offered us by a committee of review set up . 
is after the September, 1938, meetings of the American Psychological ft! 
Association. This committee came into existence as the result of Dr. li 
Sells’ suggestion at the ESP Symposium in Columbus that it would be 
advisable to present reports in a manner to permit psychologists to form | 
a more definite judgment as to the adequacy of the experimental pro- 
cedures used. Dr. Rhine asked that Dr. Sells act as chairman of such 
a committee of review and criticism of ESP papers, and a group of 
interested psychologists responded affirmatively to Dr. Sells’ request 
that they serve upon such a committee. The present editors regarded 
the value of such critical reading of reports as huge indeed, and deter- 
mined to send manuscripts submitted to this journal to Dr. Sells’ 
ee committee for reading and critical comment. The policy of this journal 
‘ will be to make the utmost possible use of the review committee, at the 
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same time holding neither the committee nor any individual member of 
the committee responsible for any shortcomings whatever either in the 
journal or in any single contribution to it. In fact, the journal can only 
proceed on the basis of a general rule current in scientific journals that 
the responsibility for facts and interpretations lies with authors and with 
them alone. This latter rule does not exempt the editors from respon- 
sibility in doing all that is possible to make certain that experimental 
conditions are actually as described and that results are actually as re- 
ported. But, as in all things human, the final responsibility lies with 
those who report what they have done in their laboratories. 

The plan to publish one long monograph and a number of short 
articles in each number does not mean that long articles are regarded 
as complete or final statements of their problems nor that short articles 
are regarded as of less value than long ones. It seems important, how- 
ever, to insist that ESP research be intensive and thorough, rather than 
extensive and superficial. In general, short articles are published for 
what they may be worth as suggestions in the direction of method, and 
will in all cases be followed, we dare hope, by technical and complete 
monographic reports. 

Readers are very strongly urged to send comments on the present 
issue to the editors or to any member of the Board of Review* and to 
help us maintain a journal which will be of serious professional interest 
to psychologists, as well as being instructive to general readers. There 
is no reason for cultism or special pleading in the present type of inves- 
tigation. Science in a democracy is the business of everyone, and efforts 
in scientific research are a fair target for anyone’s criticism. We hope 
that all experiments reported here will be criticized and, even more 
important, repeated by impartial investigators. 

G. M. 
B. F. R. 


* At present the members of the reviewing committee are: Dr. Saul B. Sells 
(Chairman), U. S. Works Progress Administration, 70 Columbus Ave., New 
York, N. Y.; Dr. Lillian Dick, Sarah Lawrence College, Bronxville, N. Y.; Dr. 
. J. Gibson, Smith College, Northampton, Mass.; Dr. E. R. Hilgard, Stanford 
niversity, Stanford, California; Dr. J. L. Kennedy, Stanford University, Stan- 
ford, California; Dr. Irving Lorge, Teacher’s College, Columbia University, New 
York, N. Y.; Dr. Louis Long, College of Physicians and Surgeons, New York, 
N. Y.; Dr. John Volkmann, Columbia University, New York, N. Y.; Dr. R. R. 
Willoughby, Brown University, Providence, R. I. 











REQUIREMENTS AND SUGGESTIONS FOR AN 
ESP TEST MACHINE 


J. B. RHINE 
Duke University 





ABSTRACT: The general requirements for an adequate ESP test machine 
are described in considerable detail. General requirements demand that such a 
machine be extraordinarily stable, of simple design, not subject to tampering, 
and capable of rapid operation. More specifically, the machine should make me- 
chanical selections, at random, of test material, and should provide ready means 
of recording results as the experiment proceeds. Other desirable, but not ab- 
solutely essential, conditions are described. Appended suggestions are made as 
to the requirements for a somewhat simpler machine than previously described. 





Ix VIEW of the great importance of developing a suitable ESP test 
machine and of the very considerable interest which has been raised in 
machines that are already in the course of development, it seems ad- 
visable to describe for the attention of those who may be interested the 
general requirements for an adequate ESP test machine, as we know 
them at present. The presentation of these requirements may enlist the 
interest of a number of persons who are equipped to offer valuable 
suggestions and, it is hoped, in many instances to invent and construct 
machines of their own design. If a considerable number of different 
designs are produced and a number of machines of different construction 
are developed, naturally we shall more quickly advance to the discovery 
of those best adapted to the purpose of the research. 

Any description of the requirements for an ESP test machine is of 
necessity relative at this stage. Very probably the first attempt at the 
use of the machine will lead to modifications of requirements or exten- 
sions which will demand consideration. The valuable pioneer work of 
Tyrrell in developing an ESP test machine (see June, 1938, Journal of 
Parapsychology) has demonstrated the difficulty of anticipating require- 
ments, and we may profit greatly by his experience. 

Again, some of the requirements stated in this account are relative 
to certain uses intended and not absolute for all. The description taken 
as a whole is an ideal from our present point of view, toward which the 
designing engineer might conceivably aim. If there are some require- 
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ments which cannot well be met, we must, of course, adjust accordingly 
to the best obtainable product. 

At the end will be found appended some suggestions for a less am- 
bitious machine, suitable for certain tests of more preliminary character, 
a machine which is much simpler and less expensive to construct. 


GENERAL CONSIDERATIONS 


1. Because of the serious disturbance to the experiment and to the 
subject’s morale which results from interruptions, it is highly important 
that the machine be thoroughly durable and dependable. More than 
ordinary stability is therefore required. 

2. Simplicity in design is of concern for three reasons: (a) It will 
contribute to essential economy. High cost of such a machine would 
greatly limit its range of use. (b) Repairing—if it comes to that— 
would of course be easier the simpler the machine. (c) Presumably 
greater stability would go with minimizing of complexity. 

3. Associated with the point of economy, it should be mentioned 
that the machine should be designed, even in the interests of research, 
to be marketable. Should selling price permit, not only are colleges and 
other research institutions possible purchasers, but wide public interest 
in the ESP research and the gamelike character of mechanized tests 
may develop a possible demand that will lead to discovery through 
amateur exploration as have certain other branches of science. 

4. A machine for such purposes would have to be subject to easy 
check in order that frequent assurance could be obtained that it was 
performing its various functions properly. 

5. The machine should be portable. It might consist of two units 
and probably would be better so. 

6. It should be impossible to tamper with this machine so as to make 
it produce extra-chance results except by means of ESP. 

7. The machine should be capable of rapid operation. One com- 
plete trial should be made in not more than two seconds and it would 
be desirable to have more speed, one trial per second for example, if 
obtainable. 

8. If necessary to have the machine operate electrically it should 
be fitted to draw power from the local power circuit (60 AC, 110V). 


SPECIAL REQUIREMENTS 


1. Perhaps the main requirement is a mechanical selection system 
which will select one of five small compartments in which objects (ESP 
symbols will perhaps be commonly used) will be placed. The selector 
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will automatically turn on a light in the compartment selected. (Should 
a non-electrical machine be devised, some mechanical way of indicating 
the selection may be substituted for the lights ; for example, a card may 
be raised or exposed within one of the compartments.) This selector 
must have a number of special features : 

(a) No sensory cues must be given the subject. It is desirable to 
have the five compartments, the selector mechanism, and whatever re- 
cording device is used to register the selection made—all in a separate 
case which can be removed to a room distant from the subject. A push 
button on the panel in front of the subject is, of course, necessary to 
energize the selection system when desired. (In some experiments it 
may be desirable for the experimenter instead of the subject to press 
the selector switch, or even to have it automatically energized at a fixed 
rate. This will call for special adaptation.) A mechanical lever would 
do for a non-electrical system. 

In case a selection mechanism could be devised which can either be 
safely regarded as soundless, or from which the sounds are of no indi- 
cation value in identifying the box selected, there will, of course, be no 
need to remove the box containing the selector system. But this would 
require strong assurance of the adequacy of the exclusion of all cues, 
auditory and visual. 

(b) Selections must represent random distribution. There must be 
no significant favoring of one compartment over the others in the total 
number of selections over a lengthy series (e.g., in 10,000 selections, no 
one box should be actually favored) nor must there be extra-chance 
distribution in, for example, avoidance of repetition or in excessive 
repetition. 

(c) It should be possible to check the distribution of selections for 
a long series not only in advance of experimental use of the machine, 
but during the period of its use. This calls either for a system of indi- 
vidual counters for each compartment selected or for a permanent record 
of the selections made. 

(d) The selector mechanism should be adequately safeguarded against 
possible tampering. It must not be possible to alter its wiring or record- 
ing so that its results will be unduly influenced. This may call for a 
system for locking the case containing it. 

The selector mechanism apparently offers the greatest range of pos- 
sibility to the inventor. In addition to the above limitations, there are 
the general considerations of expense, reliability, speed of operation, 
and others to be kept in mind. 
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2. The requirement next in importance is that of the recording of 
the results. Mechanical counters would offer one means of doing this. 
However, it would be highly desirable to have actual records of selec- 
tions and of the subject’s choice, preserving in the record the actual 
order in which these occurred. This would permit certain studies to be 
made that would be impossible on the mere record of the counters 
themselves. Some further details are in order: 


(a) If counters alone are used, the following are needed: total trials 
made (1), total hits made (1), individual compartments selected (5), 
individual choices made by the subject (5), individual compartments 
selected and hit (5), number of trials in the run (i.e., up to 25), with 
counter resetting automatically at the end of the run (1), number of 
hits in a run—this counter running up to a possible 25 and resetting at 
zero when, after each run of 25, the starter button is pressed to reset 
the machine for a new run. This calls for a total of 19 counters, two 
of them of peculiar construction requiring to be reset automatically. 

The trials-in-run and hits-in-run counters may be mounted on the 
panel before the subject in order to guide him in the course of the run. 
The individual choice counters (5) may be mounted behind the subject’s 
panel for convenience. All others should be mounted in the remote box 
if two are provided. In any case, the total trials and total hits counters 
must be securely locked to safeguard against tampering. This should 
be done so as to permit the counters to be read through windows. It is 
necessary for all the counters (with the exception of the individual 
selection counters) to be accessible to the experimenter without having 
to unlock the apparatus. These should be visible only by unlocking the 
case unless they are wired to record only after the choice button is 
pressed and the trial over. 

(b) If a recording system is used, the 15 individual counters may 
be eliminated. The other 4 would be required. In a machine of reduced 
cost, however, the total hits and total trials relays could, with some 
sacrifice of convenience, be eliminated in view of the availability of 
permanent records. 

The recording system, too, must be locked and safeguarded beyond 
possible manipulation. The recording should be on a continuous roll 
which can be officially stamped at the beginning. 


3. In addition to the general switch energizing the whole apparatus, 
there should be a switch reversing the usual order (mechanical selection 
first and subject’s choice second) making it the routine for the subject 
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to press one of the choice buttons first and then to press the mechanical 
selection button. This permits the subject to register a prediction as 
to what he thinks the selected compartment will be. This pre-selection 
procedure, as we may call it, is designed to test the hypothesis of pre- 
cognitive ESP. The apparatus will commonly be used to test ordinary 
ESP. (We may call this latter the post-selection procedure.) By means 
of a single switch, we should be able to throw the apparatus into use 
for post-selection on the one hand and on the other for pre-selection. 

4. As soon as the selector button is pressed in the ordinary post- 
selection procedure, an indicator (let us say, green) is lighted on the 
subject’s panel (as well, of course, as a bulb being lit in one of the five 
compartments). In the pre-selection procedure, this green light will 
come on when the subject makes his choice. The second step (pressing 
the selector button) will incidentally turn it off. 

5. At the end of 25 trials, another light (e.g., red) will come on 
(a trial registers and is numbered from the choice buttons pressed in 
post-selection, and from the selector button for pre-selection). This 
light will appear on the completion of the twenty-fifth trial and will go 
out when the machine is reset by a special switch for a new run. 

6. A reset switch is required to prepare the machine for the next 
run; that is, to turn off the red light indicating the end of the run and 
to reset the trials-in-the-run counter to zero. 

7. As just mentioned, it should be made impossible to register or 
record more than one choice at a trial. This may be done by making 
it mechanically (or electrically) impossible to press two or more of the 
choice buttons per trial. 

8. A switch should be installed on the subject’s panel enabling him 
to cut off the recording system as well as the total hits and trials counters 
for “off the record” runs desirable for preliminary experiments. 


ConpITIONS DEsIRABLE But Not ABSOLUTELY REQUIRED 


1. A small inconspicuous white light could be installed on the sub- 
ject’s panel which would come on when the general switch to the ap- 
paratus turned on. 

2. When the switch is pressed to reset the machine for another run, 
it would be very advantageous to have a small card released automati- 
cally stamped with the record of the number of hits made from the last 
run. This stamping and releasing would be energized by the closing of 
the reset switch. This is suggested by the public penny scales which 
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release a ticket with the weight printed on it. The advantage to the 
interest side of the research in having this device installed would be 
considerable, especially for child subjects. 


3. It would be desirable to have neat cases holding the apparatus 
and an attractive panel for the subject. This panel might be in part 
built into the door of the apparatus case which, when open, reclines down- 
ward toward the subject making a small angle with the table. Thus the 
subject’s main responses would involve easy downward pressure. The 
light, counter, and other switches could be on the upright panel in front 
of him. 


4. It is desirable to have the telegraph key type of switch for the use 
of the subject because, first, they require little pressure because of good 
leverage and, second, there is less likelihood perhaps of incomplete opera- 
tion of the multiple switch. 

5. It is suggested that the best arrangement for the choice switches 
on the subject’s panel is that of a semicircle. In the center of the circle 
the selection switch may be placed. It may be of advantage to have an 
additional selection switch which can be plugged in and moved from one 
side to the other in case the subject prefers to use his free hand on the 
selection switch, leaving one hand entirely free for the choice buttons. 
This extension would also allow the experimenter to do the selecting if 
desired. The radius of the semicircle and the circumference at which 
the choice buttons are spaced should be about three to four inches. The 
buttons should be large, about the size of the ordinary telegraph key, 
with a special button for the selection system. 


APPENDED SUGGESTIONS FOR A SIMPLE MECHANICALLY OPERATED 
ESP ApparATUS 


A less complicated apparatus might be found very satisfactory for 
certain limited test purposes in ESP work, and the greater simplicity 
and economy may encourage wider usage. The following suggestions 
are offered by way of a beginning for those readers who may be 
interested. 

A. General. A non-electrical machine is suggested, enclosed in 
locked case with glass top through which a record of the last 25 actual 
target-and-call symbols may be observed as exposed on a continuous 
paper roll. This roll is the permanent record and the immediate check 
is made through the glass window. 


B. Stimuli. The symbols are to be in five boxes, as above, and in- 
stead of lighting the box selected the card therein will be raised from an 
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inverted position to an upright angle facing the subject. The boxes are, 
of course, lightproof. Any symbols or cards may be used. 


C. Recording. Recording on the continuous roll should be accom- 
plished either by the typewriter principle (effected by the pressure of 
the choice keys) or by the adding machine plan of setting the type to be 
recorded by key pressure and actually energizing the recording by a 
lever operated by another hand. (The latter may be better since the 
mechanical selection has to be recorded also, and not before the choice 
of the subject is recorded. That is, the selection is made, but not 
recorded, and the recording apparatus is in readiness to record the selec- 
tion. This is done simultaneously with the recording of the choice.) 


D. Selector. There is needed a compact and reliable mechanical 
selector that will be free from extra-chance tendencies to favor any one 
symbol. This is likely the most difficult part of the machine because of 
the requirement of randomness. 


The selector should be worked by an arm or lever that can be at- 
tached to either side of the machine. Pressing the arm makes a selec- 
tion which raises a card in one of the five boxes to a vertical position, 
the face turned toward the subject. It remains there until a choice is 
made, the choice button releases it and it falls back into position. The 
selector arm also sets the recording device in position to record the 
selection made, this position being held until the choice button is pressed 
and the record of the selection made along with that of the choice itself. 
The entire machine should be returned to a state of readiness by the 
choice button. 

E. Regulation. A run of 25 trials should be indicated by extra 
spacing in the record tape. This can be arranged by use of a wheel with 
25 teeth running the spacer, and having a longer reach on the 25th. 

F. Safeguards. (1) There must be no way of favoring selection 
of a given symbol or box by mechanical means, such as inserting an 
instrument, tipping the machine, or any other measure available to the 
subject without breaking open the case in which the machine is securely 
locked. (2) It must be impossible to gain any advantage by pressing 
more than one choice button at a time. (3) The selector must be 
soundless or make only indistinguishable sounds. 

G. Extras, desirable but unessential. (1) If a bell can be sounded 
by the apparatus as the 25th trial is completed, it would be very ad- 
vantageous. (2) If a lock arrangement can be added, holding a cover 
over the glass window during the run of 25 trials, and releasing the 
catch when the 25th is completed, this will (a) take the place of the 
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bell (the release of the cover calling attention to the ending of the run) ; 
(b) conceal the order of selections from view during the run and thus 
make the problem of the selector somewhat easier; and (c) avoid pos- 
sible distraction from watching current hits and misses. The score for 
each run can be obtained, however, when the run is completed. The 
cover would have to be pressed down until it snapped fast before the 
next run could be started (let us say, the selector arm would not work 
with the cover up). (3) Mechanical counters of total trials and total 
hits could be inserted—also, of course, individual selection and choice 
and hit counters for each symbol and each choice button. But with a 
complete tape record only totals (trials and hits) would be of importance. 








A MACHINE FOR RESEARCH IN EXTRA-SENSORY 
PERCEPTION 


ERNEST TAVES 





ABSTRACT: Though granting that hand-shuffling can properly be used for 
many ESP procedures, there are various situations in which automatic prepara- 
tion of material to be used in ESP ‘experiments is desirable. A machine has been 
devised which, by means of remote control, rotates so as to set up a set of 150 
stimulus objects to be guessed. 

The machine consists of a large tray divided into 150 small compartments— 
six rows of twenty-five. Each compartment can contain a disc numbered in one 
way on one side, in another way on the other side, or any regular solid polygon, 
such as a die. The tray is electrically rotated, and is brought to rest in a hori- 
zontal position, a photograph of the set-up being taken by remote control. 

Preliminary data suggest that the dice and discs behave in a manner not 
normally predictable, and that the machine can properly be used for setting up 
a target normally known to no one. 





F OR SOME TIME researchers in extra-sensory perception have realized 
that it would be of value to have stimulus objects prepared mechan- 
ically in a random manner. The purpose of this paper is to describe 
such a machine, construction of which has recently been completed by 
the writer. 

In another paper’ in this issue of the Journal is described consider- 
able research using various types of cards as stimulus objects. The 
machine to be described here was constructed in such a manner that a 
comparable set-up of stimulus objects could be prepared without directly 
involving any human agency. That is, it was desired to construct a 
machine which would prepare six sets of twenty-five objects, the ob- 
jects being such that the odds against their being “guessed” correctly 
were in some cases even, in some cases four to one. For the fifty-fifty 
odds, flat discs or chips may be used. In this case either face of the 
disc may be “up,” and the subject is required to guess which face is 
actually uppermost, the disc lying flat. There may be a circle on one 
side of the disc and a cross on the other, for example. 

For the situation in which the odds are longer dice may be used, in 


* Murpnry, G. and Taves, E., Covariance methods in the comparison of extra- 
sensory tasks, this issue, p. 38. 
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which case the subject is required to guess which of the six faces of the 
die is uppermost. This gives odds of five to one rather than of four 
to one as with the standard ESP cards, but this presents no serious 
difficulties. Other regular solid polygons could be used. 


In first planning the machine, then, the materials decided upon were 
flat discs and dice. The next problem was concerned with the manner 
in which these materials were to be “shuffled.” It is reasonable to as- 
sume, barring mechanical perfection, that if either a die or a flat disc is 
placed in a box, and the box is rotated for a short time, allowing the 
object inside to bounce freely from side to side, the face which will be 
uppermost when the box is stopped will be unpredictable. Acting upon 
this assumption a tray was constructed which met the necessary require- 
ments. 

The tray consisted simply of six rows of twenty-five compartments, 
the dimensions of which were 134” x 134" x 2%". The base of the 
tray was a 4 x 11%’ rectangle of 34” plywood, and the sides of the tray 
were constructed of the same material. The tray was divided into six 
longitudinal rows by partitions of wood, and each of the six rows was 
divided into twenty-five compartments with the use of thin veneer par- 
titions, which were inserted in slots in the longitudinal partitions. A 
polished plate glass lid, 5/16” thick, was fastened over the tray with 
3%" angle brass. This angle brass was, on one side of the tray, readily 
detachable, permitting easy removal and replacement of the lid. 

The next problem was that concerned with rotating the tray long- 
itudinally. This involved the question of rate of rotation. If this rate 
were too slow, the objects would simply slide from side to side, and 
would not be adequately bounced ; if the rate were too fast, centrifugal 
force would hold the objects flat against the outermost side of the com- 
partment. It was found that a rate of rotation between thirty and forty- 
five r.p.m. would meet the requirements. Thus a ratiomotor was used, 
which rotated the tray at thirty-eight r.p.m. The ratiomotor used, made 
by the Boston Gear Works, was simply a unit consisting of an electric 
motor and a gear reduction system. The motor was of the sixty-cycle, 
single-phase, ac type, operating on 120 volts. 


The ratiomotor was mounted on a 1%’ x 1%’ plywood platform, 
and the platform was mounted on one end of a six-foot wooden base. 


Shafts of 5%” steel were attached to each end of the tray. One shaft 
ran through an adjustable bearing and a semi-flexible coupling to the 
ratiomotor, the other through a pillow block mounted on the base, oppo- 
site the motor platform. Thus the tray was rotated longitudinally. 
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Next came the problem of controlling the rotation adequately. Since 
operation was to be by remote control, it was necessary automatically 
to stop the tray in a horizontal position. One way of doing this would 
involve the insertion of a clutch between the motor and the tray, so 
that the tray could be disengaged, and brought into a horizontal position 
by a counterweight on the bottom of the tray. The clutch would be. 
necessary since the gear reduction system, being of the worm type, did 
not permit reverse rotation; otherwise the tray, provided a counter- 
weight were properly adjusted, would swing into a horizontal position 
immediately after the motor current were cut off. 

The clutch method proved to be impracticable, and it was necessary 
to use other means. An electrical commutator was therefore used. Two 
brass discs, 44%" in diameter, were placed on the shaft between the 
motor and the tray. One of these discs was continuous, one discon- 
tinous. The two discs were so placed as to dip into open mercury cups, 
immediately below. One disc would then be in the mercury contin- 
uously, the other intermittently. Ifthe mercury cups were then made the 
poles of an electrical circuit, current would pass through the circuit 
only when both discs were in the mercury. Thus the commutator 
would serve as a simple switch, either to start or stop the motor. This 
electrical system is shown in Figure 1. 
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Ficure 1. Electrical circuit showing wiring of motor. 


S; is a switch operated by the experimenter in a distant room. Sg 
is the commutator switch. When either switch is closed the motor will 
operate. The “break” section of the commutator was of an arc five 
degrees less than the arc through which the tray coasted after the cur- 
rent was cut from the motor. Thus when the experimenter’s key was 
released the discontinuous commutator disc would be either in or out 
of the mercury cup. If it happened to be in, the motor would run until 
the disc left the mercury, breaking the circuit. If it happened to be out 
of the mercury the motor would coast until it dipped in again, would 
run until it dipped out, and would then coast to a stop as before. Or if 
the experimenter’s key was released just as the discontinuous disc left 
the mercury the result would be the same. The five degree allowance 
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in the discontinuous disc was necessary to prevent the circuit from 
being continually opened and closed. 


The assumption of the writer at this point was that the distance 
which the motor coasted after the current was cut off was constant. 
Thus assumption was not justified ; the distance which the motor coasted 
was not constant, due to the varying viscosity of the lubricant in the 
gear reduction system as the motor gradually became warm. That is, 
the friction in the system was not constant. Therefore it was neces- 
sary to add to the system an additional, constant, friction of such mag- 
nitude as to blanket the variability previously present in the system. 


For this purpose a flat pulley was placed on the shaft between the 
motor and the tray, and a leather belt was run over this pulley and 
down to the base, where it was rigidly attached. This additional friction 
lessened the distance which the tray coasted after the motor stopped, 
and also lessened the variability of this distance, so that the final stop- 
ping position of the tray was subject to a variability of less than ten 
degrees. 


At this point a small empirical test was made, and it was found that 
although the tray stopped in the same position each time, the discs did 
not always lie flat on the bottom of the compartments ; they sometimes 
would lean against the side of the compartment. To avoid this the com- 
mutator was adjusted so the tray stopped about fifteen degrees beyond 
the horizontal position. Also, the bottoms of the compartments were 
lined with celluloid film in order to reduce the friction between the 
discs and the bottom of the tray. 


It was also found that the dice did not bounce from side to side, but 
simply slid from one side to the other. Thus it was necessary to place 
small pads of soft sponge rubber on one side of each of the compart- 
ments in which dice were to be used. These pads caused the dice to 
bounce freely and unpredictably. 

In addition to preparing the stimulus objects by remote control, it 
was desirable to record the entire set-up in the same way, so that no 
person would have to be in the room with the tray for any reason what- 
ever. For this purpose it was necessary to use a camera which would, 
in a single operation, expose the film and move the film into position 
for the next exposure. The Sept camera, of obsolete French make, was 
found to be adequate for this purpose. This is a movie camera which 
can be used on a single exposure basis. 

The camera was mounted above the tray, the plunger being operated 
by a heavy solenoid, controlled from a distant room. One loading of 
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the camera permits as many as 250 exposures. It is thus possible for 
an experimenter to go through 250 experiments, each consisting of 150 
trials, without entering the laboratory room in which the apparatus is 
placed. 

Illumination for the photography of the set-up was provided by two 
No. 2 Mazda Photoflood lamps in reflectors, mounted above the tray. 
A sketch of the entire apparatus is shown in Figure 2. 
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Ficure 2. 


The photographic aspect of the recording is desirable for several 
reasons. For experiments to be conducted on the sole responsibility of 
the experimenter and his immediate assistants, the negatives need be 
used only, a projector permitting easy reading of the impressions. And 
for co-operative experiments in which other members of his own uni- 
versity staff may watch from beginning to end every phase of the ex- 
perimental work, examining the results at leisure afterwards, and for 
inter-university long-distance work involving the exchange of records 
by various experimenters, reliance must be placed upon photographic 
prints. Also, the photographic field can include an electric clock and a 
calendar card, and the postmark of mailed photographic prints can show 
the last moment at which a particular experiment could have been 
performed. 

One of the advantages of the machine described is that a great 
variety of materials can easily be used, it being necessary only to remove 
the glass lid of the turntable to effect immediate change of test objects. 
It is possible, for example, to use material of emotional quality. Discs 
may be used, on one side of which is a simple drawing, such as a tragic 
profile, there being on the other side of the disc a comic profile. Various 
types of colored discs may, of course, be used, as well as great variety of 
types of symbolic material. 
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Another advantage is that the machine provides a means for self- 
testing. While the laboratory in which the apparatus is placed remains 
locked, the subject can set up his own targets by remote control without 
necessitating the presence of an experimenter. The records of the sub- 
ject’s guesses can be placed in the hands of the person responsible for 
the experiment before anyone is admitted to the laboratory in which is 
placed the machine. 

An empirical control of some length has already been made, and no 
extra-chance factors are apparent. A further control is being made, 
and results will be fully presented in a later issue of the Journal. 
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REPLY TO CRITICS OF THE STANFORD 
EXPERIMENTS ON THOUGHT- 
TRANSFERENCE 


JOHN E. COOVER’ 
Late Fellow in Psychical Research, Stanford University 





ABSTRACT: Critics of Coover’s 1917 monograph, reporting research on 
thought transference (Stanford University), have stated that conclusions other 
than those drawn by Coover are indicated by the data. Coover points out that 
these criticisms are not valid because of fallacious selection of particular experi- 
ments, uncontrolled sources of error in the original research, and lack of regard 
for negative results obtained in experiments in which the subjects were of the 
“gifted” type. 





I N THE MONOGRAPH? that has evoked protests from capable critics 
there are several sections that relate to the problem of thought-trans- 
ference: (1) 1000 trials with one subject on the guessing of lotto block 
numbers, (2) 10,000 trials with normal subjects and 1000 with “sensi- 
tives” on the guessing of 40 playing cards, (3) 2400 trials on “the 
feeling of being stared at,” (4) 15,000 trials on the influence of sub- 
liminal visual and auditory cues upon guessing, (5) data on mental 
habits, which play a part in “telepathy” because of community of ex- 
perience, shown in guessing playing cards and numbers, in reading 
scientific instruments and in estimating magnitudes, (6) an exhibit of 
data on inductive probability showing safety of reliance on the theory 
of probability in evaluating the experiments, (7) a section on infinitesi- 
mal probability showing that highly improbable events may be expected 
to occur in daily life, and (8) a critique of the evidence offered to 
prove the reality of telepathy. The conclusions were in general nega- 
tive with respect to thought-transference, but positive with respect to 


* Due to the author’s untimely death, this manuscript was left unfinished. I 
have added only the last part of the analysis of Thouless’ criticisms and the 
concluding statements. The responsibility for these statements rests solely with 
me, although we were in essential agreement on them.—John L. Kennedy, Fellow 
in Psychical Research. 

* Coover, JoHn E. “Experiments in Psychical Research at Stanford University” 
Stanford University Press, 1917, xxiv + 641. (Now out of print.) 
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the “subjective” experience of telepathy in the laboratory, to the effect 
of subliminal cues in guessing, to the effect of mental habits and to 
the trustworthiness of theoretical probability in the evaluation of card 
guessing experiments. 

The criticisms which I shall endeavor to answer apply to the sec- 
tion of experiments with 100 normal subjects on the guessing of 40 
playing cards. These criticisms are based upon recomputations of re- 
sults abstracted from the tables of data exhibited in the monograph, 
and tend to controvert the negative conclusions reached in the mono- 
graph. 

Dr. F. C. S. Schiller, in his review of the monograph, says of the 
card-guessing experiments, 

The experiments constituted a good test of the hypothesis that the capacity 


for receiving telepathic impressions is faintly diffused throughout the human 
race (p. 262). 


He examined the final totals of the tables: 











Numser RIGHT ON 
Guesses 
Card Color Number Suit 

Obtained 
Card-not-Imaged.......... 141 2491 488 1252 4865 
Card-Imaged.............. 153 2556 538 1344 5135 

Expected by Chance 
Card-not-Imaged.......... 122 2433 487 1216 4865 
Card-Imaged.............. 128 2568 514 1284 5135 

Deviation 
Card-not-Imaged.......... 19 58 1 36 
Card-Imaged.............. 25 11 24 60 
Chance Limit............. 47 150 90 130 5000 




















and observes, 


The results taken in bulk were negative in the aggregate, and so did not sup- 
port the belief in faint degrees of telepathy. . . . There was a slight excess 
over the most probable number in both the Card-not-Imaged and Card- 
Imaged series, and a slight difference in favor of the Card-Imaged experi- 
ments in which telepathy might conceivably be operative; but it was not 
sufficient to warrant any inference as to the reality of telepathy, as it was 
not above the limits of chance variation. 


And he notices that if the totals of the separate thousands of guesses 
are examined, some of them in the Card-not-Imaged series 


™ * Proceedings of the Society for Psychical Research (London), 1918, 30, 261- 
3. 
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might have been taken as better proof of telepathy than any in the Card- 
Imaged experiments (pp. 262-3). 


Thus far the critic is in agreement with the text. 


When, however, we come to test the hypothesis that telepathy occurs in cer- 
tain (rare) subjects, the interpretation of these figures is by no means as 
clear. For it then becomes legitimate to argue that some only of the reagents 
had faint telepathic capacity, and to select their answers for separate analysis. 
... Hence we may select 14 reagents . . . and tabulate their results. 


The totals of this tabulation of 14 sets of Card-Imaged experiments 
and of another tabulation of an equal number of selected results from 
the Card-not-Imaged series, are as follows: 











Guesses RiGHT ON 
Guesses 

Card Number 

eee a 54 119 711 

Card-not-Imaged............. 49 91 690 
Card-Imaged................. 7.6% 16.7% 
Card-not-Imaged............. 7.1% 13.2% 
Chance Limit.......... 5.0% 14.8% 














Concerning the Card-Imaged successes, Schiller says, 


There was therefore a considerable excess over the ordinary chance distribu- 
tion. But was it too great to be ascribed to chance, and must it be attributed 
to an unknown cause (“telepathy’’) ? 


He refers to the table of percentages and learns that the per cents 
right exceed the chance limit, and continues, 
This is on the face of it a considerable excess, and if the method of reason- 


ing is sound, would appear to prove some degree of telepathy in the 14 
selected subjects (p. 263). 


It should be noted that there appears to be some question in Dr. Schil- 
ler’s mind as to whether this procedure of recomputation is sound. 

Dr. Schiller then turns his attention to the Card-not-Imaged re- 
sults of the 14 subjects who got three or more cards right, and notes 
that on the “card right” his computation shows them also to exceed 
the chance limit, and on number right, closely approach that limit. In 
fact, there is little extra-chance cause to be granted the Card-Imaged 
group that cannot also be granted the Card-not-Imaged group. Does 
it seem to be telepathy or clairvoyance that may be the extra-chance 
cause ? 
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Dr. Schiller notices that in the two groups of 14 best guessers five 
subjects are common and he computes their results separately : 




















Guesses RIGHT ON 
Guesses 
Card Number 
Card-Imaged................. 19 45 240 
Card-not-Imaged............. 22 41 260 
nce cnsaceeua 7.9% 18.7% 
Card-not-Imaged............. 8.5% 15.8% 
Ea 8.2% 17.2% 500 
Chance Limit............ 5.5% 15.7% 








The percentages .. . are very markedly higher than the maxima at- 
tributable to chance. . . . These figures therefore distinctly point to some 
source of rightness beyond chance in these cases. As it occurs in the Card- 
not-Imaged series, it cannot be of the nature of conscious Thought-Trans- 
ference. But may it not be due to a sort of “lucidity” or clairvoyance in 
the subject? (p. 264). 

Dr. Coover is hardly entitled to deduce from his data that “no trace of 
an objective thought-transference is found as a capacity enjoyed in per- 
ceptible measure by any of the individual normal subjects” (p. 124). He 
should recognize not only, as he does (Coover, p. 65), that his “control” 
(Card-not-Imaged) experiments assumed the non-existence of any “lucidity,” 
but also that this may have been wrong, and that they may not have been 
“pure guesses,” and endeavor to experiment further with the abnormal co- 
incidence of good guessers successful in both series (p. 265). 

If the excess of right answers continues (with these subjects), he will 
be able to establish the existence of telepathy on a statistical basis. Experi- 
ments with these five good normal subjects would appear on the evidence to 
be more promising than with the 10 “sensitive,” whose 530 Card-Imaged 
experiments only showed a comparatively slight excess over probability, 
and whose guesses in Card-not-Imaged experiments gave no hint of lucidity 
(p. 265). 


(Is it not curious that these selected normal subjects should show more 
promise for lucidity than do mediums and “sensitives” who are reputed 
to have it ?) 

In a footnote (p. 264), Schiller draws attention to the fact that while 
in the two groups of 14 best guessers, five subjects are common to 
both sets, in two groups of worst guessers, 24 subjects in the Card-not- 
Imaged experiments and 20 in the Card-Imaged experiments, only three 
subjects are common to the two groups. The best guessers tend more 
than the worst guessers to express their trait in both the Card-Imaged 
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and Card-not-Imaged experiments. The best guessers showed results 
above chance expectancy in both groups; the worst guessers in the 
Card-Imaged experiments showed about chance results in their Card- 
not-Imaged experiments, but the worst guessers in the Card-not-Imaged 
experiments showed above-chance expectancy in their Card-Imaged ex- 
periments. Thus this last group of worst guessers shares with the best 
guessers some benefit from imaging the card. 

We may carry further this method of recomputation of the results 
of selected groups: 

1. There were 20 worst guessers in the Card-Imaged experiments, 
who, in sets of approximately 50 guesses, got zero cards right. (This 
was the basis of their selection.) Total number of trials is 1026. 


Card right = 0 n =1026 
np expected = 26 p = 025 
Difference = 26 q = 975 
D/e = §.12 oe? = 25.0 
Chance limit = 4.24 


Here the difference divided by the standard deviation (7) of the bi- 
nomial distribution is 5.12, which exceeds the adopted chance limit of 
4.24. Is this proof of reversed telepathy? If the deviation is below 
the chance limit, the guesser must have known the card in order to 
miss it. 

2. There were 24 worst guessers in the Card-not-Imaged experi- 
ments : 


Card right = 0 n =1143 
np expected = 28.6 p = 025 
Difference = 28.6 q = 975 
D/e = 541 e = 27.9 
Chance limit = 4.24 


Again a result which is over the chance limit. Is this reversed clair- 
voyance? 

3. Among a considerable number of sets of drawn cards matched 
with the preceding guess there were seven sets in which three or more 
cards were right. By selecting these for independent treatment we have 
the tabulated computation from p (probability of a card right) — .025, 
n (number of guesses) — 374. 


Card right =24 


np expected = 9.35 
Difference = 14.65 
D/e = 485 
Chance limit = 4.24 
o = 912 
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This extra-chance result is significant because it exceeds the limit of 
chance fluctuation, set high for logical reasons and for convenience at 
4.24 times the standard deviation. If the separateness of computation 
is sound, we seem to have proof of prescience, prediction, or prophecy. 
The relation is between the guess and the following drawing of a card 
from a newly shuffled pack. 

4. A still more startling result may be shown from the separate 
treatment of the figures obtained in matching the cards drawn in one 
set of approximately 50 with those drawn in another; in four such 
sets there were 4 cards right in each set. 


Card right =16 n = 197 
np expected = 4.93 p =.025 
Difference = 11.07 q =.975 
D/e = 5.03 
Chance limit = 4.24 
o2 = 4.80 


Were this extra-chance result to be warranted, we should have to search 
for some demon that correlates drawings of cards from immediately 
shuffled packs manipulated by different persons at different places and 
times or for mischance in tabulating figures in tables or in marking or 
counting coincidences ; either demon, or (2) experimental, (3) clerical, 
(4) statistical, or (5) computational error. (I have numbered these 
alternatives because I believe them to be often neglected when they are 
potent for accumulating spurious coincidences. ) 

5. Suppose we now test this method of recomputing results of a 
group selected from a distribution by selecting for separate analysis 
only the results of the 8 subjects who got 3 cards right in the Card- 
Imaged experiments. 


n= 399 
Card right =24 Number right = 67 
np expected = 10 np expected = 40 
Difference =14 Difference =277 
D/o = 4.50 D/e = 4.50 
Chance limit = 4.24 Chance limit = 4.24 
o2 = 973 o2 = 35.90 


In this last sample, the number of cards right and of number right is 
also above the limit of chance yet the probability of getting 3 cards 
right is .093 (the fourth term in the expansion of (q+ p)5®) and 
.093 & 100 (sets of 50 guesses) = 9.3. On the basis of pure chance, 
9 cases of 3 cards right are expected in the distribution. In our dis- 
tribution of actual results in the Card-Imaged experiments we obtained 
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8 such cases (one less than chance expectation), but when we select 
those cases and recompute them separately, we seem to have statistically 
significant proof of an extra-chance cause. 

It is evident that the separate treatment of a part of a distribution 
calls for a special method. Case 5 shows that part of a chance dis- 
tribution may be made to show cause apart from chance. All one 
needs to do is to select sufficient data removed well from the mean of 
the distribution and he can “prove” the presence of any cause he has 
an interest in promoting. 

Selection for separate treatment should not be made from the fig- 
ures in the distribution; it should be made from the conditions of the 
experiment. Or, if selection is made on the basis of figures in the 
distribution, it should be for the purpose of going back to the data to 
seek for conditions that may be responsible for the peculiar nature of 
these figures, rather than for putting them through a statistical mill 
that is no more applicable to them. 

This demonstration of the fallacy of “selected” cases in statistical 
populations is made for the benefit of those who not only overem- 
phasize but misinterpret Dr. Schiller’s criticism. Dr. Schiller, at the 
beginning of his consideration of the hypothesis that telepathy occurs 
only in certain rare persons, says, 

It then becomes legitimate to argue that some only of the subjects had 


faint telepathic capacity, and to select their answers for separate analysis 
(p. 263). 


And, after exhibiting that the results of his recomputations were above 
the chance limits, he says, 


This is on the face of it a considerable excess, and if the method of rea- 
soning is sound, would appear to prove some degree of telepathy in the 14 
selected subjects (p. 263). 


He notices that “card right” by the best 14 guessers in the Card-not- 
Imaged experiments is also above the “chance limit,” and then turns 
his attention to the 5 subjects common to the two groups; after com- 
bining their Card-not-Imaged and Card-Imaged scores, he finds them 
“markedly higher than the maxima attributable to chance,” and says 
these figures therefore distinctly point to (not definitely prove) some 
source of rightness beyond chance in these cases. 


May it not be a sort of lucidity in the subject? Doctor Coover is hardly 
entitled to deduce from his data that “no trace of an objective thought- 
transference is found as a capacity enjoyed in perceptible measure in any 
of the individual normal subjects” (p. 124). He should recognize not only, 
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as he does (Coover, p. 65), that his control “Card-not-[maged” experiments 
assumed the nonexistence of any lucidity, but also that this may have been 
wrong, and that they may not have been pure guesses. 


He does not say that there appear to be excesses here beyond chance 
guessing and that further experimenting with the best guessers, if they 
hold up, would “establish the existence of telepathy on a statistical 
basis” (p. 265), and might be more profitable than further work with 
sensitives in search of lucidity. 

So far, then, (1) Schiller does not demur at the high critical ratio 
(D/*= 4.24) chosen for the limit of chance variation. (2) He agrees 
that the totals of all the experiments do not pass this limit and that 
therefore the hypothesis that telepathy as a capacity existing faintly 
in all men is not supported; he draws attention to excesses beyond 
chance expectation in the results of selected best guessers that “dis- 
tinctly point” to some source of rightness beyond chance in these cases. 
(3) He thinks the experimenter was “hardly entitled” to say that no 
trace of an objective thought-transference as a capacity enjoyed in 
perceptible measure by any of the individual normal subjects was found, 
and (4) he thinks that guessing in the Card-not-Imaged experiments 
was not pure guessing. 

The monograph conclusions were: (1) No telepathy as a common 
capacity was found because the general totals did not exceed the chosen 
chance limit; (2) No telepathy as a capacity of single individuals was 
found because no totals of single sets exceeded the limit of chance. The 
same conclusions could have been deduced with reference to lucidity. 
After Schiller’s criticism, both of these conclusions still stand. How- 
ever, something more could have been done with the excesses that 
challenged Schiller’s attention ; these are sub-critical, below the limit of 
chance variation; while they count zero for telepathy, which is re- 
quired to pass the limit of chance, they may count significantly toward 
establishing any known causes. But these may wait till their entrance 
in the next review to be examined. 

Dr. Robert H. Thouless, of the Department of Psychology at Glas- 
gow University, in his review* of Professor Rhine’s monograph® takes 
note that Rhine quotes, among other investigations, 


that of Coover as providing positive evidence of extra-sensory perception. 
Since it is a common opinion that Coover’s results were entirely negative 
and show nothing but chance distribution, I have thought it worth while 

*Dr. Rhine’s recent experiments on telepathy and clairvoyance and a recon- 


sideration of J. E. Coover’s conclusions on telepathy. Proc. Soc. Psychical Res.. 
43, 24-37, 1935. 


5 “Extra-Sensory Perception,” Boston Soc. Psychical Res., 1934, pp. 169. 








REpLy TO CRITICS OF THE STANFORD EXPERIMENTS 25 


to re-examine Coover’s figures and will discuss these before proceeding 
with Rhine’s own work. Rhine is undoubtedly right in saying that Coover’s 
results actually show strong evidence against chance. There seem to have 
been two reasons why Coover himself drew the opposite conclusion: first, 
he adopted an absurdly high limit for the deviation from mean expectation 
which might be attributed to chance, and, secondly, he did not consider 
the possibility that clairvoyance might be active where telepathy was im- 
possible in the Card-not-Imaged series® (p. 25). 


Dr. Thouless also inspected the general table presented first above, 
and stated that 


The general tendency of both of these Card-not-Imaged and Card-Imaged 
series is clearly to exceed mean chance expectation, and in approximately 
equal amounts. Coover concludes that since the factor of telepathy cannot 
be present in the first series, the approximate equality of the two groups is 
due to the fact that the deviations of both are due to chance. A safer con- 
clusion would seem to be that if any factors are present causing deviation 
from expectation, these are operating in approximately equal amounts in 
the two conditions of experimentation. At any rate, we shall be justified 
in lumping the two groups together for statistical consideration. For the 
remainder of the discussion of these results, I shall do this since it will 


give us the advantage of the higher significance to be obtained by larger 
numbers (p. 26).7 


First, let us consider the reasons for adopting the “absurdly high” 
limit of chance expectancy. In a footnote on page 85 of Coover’s mono- 
graph, we find the following explanation : 


Although there are certain restrictions to the application of this formula, 
such as when m (the number of experiments) is small, or when p is very 
small, its use is peculiarly applicable to the type of data with which we 
are dealing, and is not unsupported by approved statistical methods already 
used in the field of psychical research. E., in his “Analysis of Mrs. Ver- 
rall’s Card Experiments” (Proc. Soc. Psychic. Res., 1895, 11, 193-197). 
Mr. C. P. Sanger used the following formula for the limit of chance de- 
viation from the probable number of occurrences: 
K=3y2 - (1-q)mq 
in which g is probability of occurrences and m is number of experiments. 


By appropriate transformations of symbols, this formula reduces to the 
L=vV2-3¢ 
used by Coover to evaluate his card guessing results. 
Another hint as to the reasons behind Coover’s selection of a high 
limit of chance variation is contained in a letter written to Mr. Whately 
Carington in 1938. To quote: 


_*See Coover (p. 65) for a discussion of assumptions in the Card-not-Imaged 
series. 


7 Coover’s manuscript ended here. 
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It is agreed that if there is awareness of objective facts or events, without 
the functions of the organs of sense, it is of great importance to establish 
that phenomenon scientifically (i.e., by experiment). Also agreed that no 
negative judgment is justified now or ever. But what seems not to be 
agreed is that the history of the experimental work on the problem suggests 
that there is a very small probability that the hypothesis is true, and that 
therefore a high critical ratio for the limit of chance deviation is a logical 
necessity. 


Professor Thouless does not seem to reject this logical assumption as 
necessary proof of telepathic transmission. He does object to Coover’s 
conclusion that the results were explainable on the basis of chance varia- 
tion. He says, 


The difference between observation and mean chance expectation of the 
number of cards guessed altogether right by Coover’s subjects shows a 
probability of chance occurrence very much below this limit (50-1. D/e 
2.05), being 200-1 (D/e 2.58). The existence of some factor favoring cor- 
rect guessing of the card is strongly indicated. It might be objected that 
any form of extra-sensory cognition is @ priori so improbable that we shall 
be right to insist on a much more severe criterion of significance than we 
should need, let us say, if we were trying to investigate the difference in 
fertility of manured and unmanured fields. To this objection there are 
two replies. First, the question at issue is not, at the moment, whether 
or not extra-sensory cognition occurred amongst Coover’s subjects but 
whether there was a factor in his experiments favoring correct guessing 
(such a factor might be some unnoticed error of method). There seem to 
be no grounds for regarding the presence of such a factor as very improb- 
able. A much more important consideration, however, is that if we are 
convinced of the a priori improbability of extra-sensory cognition, that will 
be a sound reason for accepting the indications of a 200-1 odds against 
chance with less conviction than we should otherwise feel; it is no evidence 
at all for regarding heavy odds against chance as evidence in favor of the 
operation of chance (p. 27). 


The cogency of this criticism is recognized. It remains, however, 
to return to Coover’s description of method in order to evaluate the 
200-1 odds. The control of these experiments may be questioned on 
two points, (1) independence of recording and (2) use of minimal 
and subliminal auditory cues. 

In order to eliminate completely errors in recording at the time of 
the experiment which are due to knowledge of the correctness of the 
guess, the recording of the cards and the subject’s guesses should be 
done separately and independently. Coover states: 


The experimenter with a watch before him, (1) shuffles the deck of forty 
playing cards (the face cards being discarded), cuts the pack, and holds 
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the cards concealed; (2) shakes the dice-box, to determine a control or 
regular experiment, and, if the latter, the form of content the latter is to 
have in his mind; (3) if a regular experiment, he turns over the pack, 
exposing to his view the under card, taps once to signal the reagent that 
the experimental period begins, holds mental content of card and wills 
the content to be projected into the mind of the reagent, and, after 15 to 
20 or more seconds, taps twice to signal the close of the interval. After 
he notes that the reagent has recorded his guess,8 and has turned to his 
introspections, he records the color, number and suit of the card and the 
number of the die-spot which conditioned the form of the experiment (as, 
R5H 1, for Red, Five of Hearts, Die-spot 1—i.e., precisely the same form 
as the regular, except that the card remained unknown until the reagent had 
recorded his guess (pp. 53-54). 


These conditions for recording would have been completely controlled 
if the agent could not have seen the subject while the latter was record- 
ing his guess. 

Referring back to the quotation on method given above, it is ap- 
parent that the experiments were carried out in the same room. Al- 
though 847 trials were collected with the agent at distances of 1, 2, 3, 
4.6, 6, and 10 meters from the percipient, no differential effect to dis- 
tance was discovered in the analysis of this series. It is possible, how- 
ever, that occasional involuntary auditory cues may have been operative 
in producing small excesses over chance expectancy when the results 
of all the 5,135 Card-Imaged trials were summed. Coover (pp. 452- 
460) describes methods and exhibits negative results for telepathy or 
any other factor besides chance variation when conditions of the ex- 
periment were such as to exclude sources of error. These experiments 
were carried out with “sensitives” who claimed to have the ability to 
obtain information by telepathy. 

The following conclusions may be drawn from this discussion : 

(1) Critics of Coover’s 1917 monograph have confined their atten- 
tion to the card guessing series with normal subjects, attempting to 
show that selected subjects from this group of 100 gave evidence of 
ability to produce extra-chance scores. In his discussion of Schiller’s 
criticisms, Coover has demonstrated the statistical fallacy in such se- 
lection. 

(2) The critics have not given due regard to the negative results 
in his tests, using 10 professional and private “psychics,” of the hy- 
pothesis that only a few selected cases have telepathic ability. 


* Italics mine. J. L. K. 
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(3) Uncontrolled sources of error may explain the positive devia- 
tions on whole card right in the “normal series” of card guessing re- 
sults. Coover’s experimental conditions did not completely eliminate 
the possibility of errors in recording and involuntary auditory cues. 

(4) These considerations do not support statements concerning 
Coover’s “positive” findings for telepathy and clairvoyance which are 
appearing in the modern literature of “Extra-Sensory Perception.” 











ADEQUATE EXPERIMENTAL TESTING OF THE 
HYPOTHESIS OF “EXTRA-SENSORY PER- 
CEPTION” BASED ON CARD-SORTING 





ABSTRACT: Some psychologists reject the conclusions of Rhine and other 
authors favorable to the hypothesis of extra-sensory perception, since their con- 
clusions have been drawn from data without full description of the conditions 
under which the data were obtained. A program of definitive nature is described. 
Suggestions are given regarding general procedure, safeguards against illicit 
cues, the way in which cards are handled by subjects, scoring and keeping of 
records, quantitative specifications, selection and orientation of subjects, selec- 
tion of assistants, general superintendence of the experiment, protection of ma- 
terials and records, and elaboration of obtained data. 





Dz J. B. RHINE and certain other authors have drawn, from data 
derived from card-sorting, conclusions which are favorable to the hy- 
pothesis of “extra-sensory perception.” Many psychologists have re- 
jected these conclusions, alleging that the data, as presented, do not 
warrant the conclusions drawn, and pointing out the principle, gener- 
ally accepted by experimental psychologists, that valid conclusions can- 
not be drawn from data in isolation, but can be drawn only from data 
in conjunction with the conditions under which the data have been ob- 
tained. As a result of these challenges, there has come from several 
quarters a request for the outlining of a program of research which 
would, if carried out, be acceptable to Dr. Rhine’s critics, and from 
which might be expected results of a definitive nature. 

In presenting the program which follows, we are attempting merely 
to apply to a specific problem the procedures, methods, and techniques 
which are familiar to, and accepted by, psychologists trained in the 
psychological laboratory, and accustomed to the requirements of rigid 
experimental work. 

It must be emphasized that in no program is it possible, in advance 
of operations under the program, to cover all precautions, or to meet 
all contingencies which may arise, and therefore the provisions of the 
several sections of this research program may not be sufficient. It 
would, for example, be impracticable in the present outline to prescribe, 
for all possible situations, ways and means of excluding sensory clues 
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or cues, since the range of possibilities is large. It is necessary, there- 
fore, that there be the most competent possible supervision, as indicated 
in Section IX below. 

It must further be emphasized that the various stringent precau- 
tions prescribed in the several sections of the program involve no im- 
plications that previous experimenters failed to take such precautions 
as were deemed by them reasonable and proper, or that there was any 
lack of good faith on the parts of experimenters or subjects. The tech- 
niques and precautions herein specified are presented solely with a view 
to the preclusion, in so far as may be possible, of all subsequent criti- 
cism. 

The experimental specifications here presented apply to a very spe- 
cific problem, and obviously cannot be transferred, simpliciter, to a dif- 
ferent problem. It is possible to furnish, within a small number of 
pages, specifications for a limited problem. To furnish specifications 
applying to all problems in the field would involve the writing of a 
manual on experimental psychology. 

The problem has been limited in such a manner as to make the pro- 
cedure involved correspond, as nearly as possible, with the procedure 
which appears to be the most widely employed by those who have made 
card tests of alleged “extra-sensory perception.” The limitations of 
primary importance are: (1) as regards basic materials; to the use of 
the cards commercially sold as “ESP cards,” or cards essentially similar 
in design and construction. (2) as regards method; to the sorting of 
the cards in the attempt to distribute them in stacks corresponding to 
the symbols borne on the faces of the cards. (3) as regards plan; 
to the deriving, for final comparison, of average scores of success or 
hits. (4) as regards the element of chance; to the condition under 
which the subject, at any time during a test, knows what disposition he 
has made of cards up to that point. 

We may point out, that for experimental tests (1) with other ma- 
terials; (2) with naming of cards instead of sorting; (3) with pur- 
poses other than the securing of average scores; or (4) in which the 
subject, in assigning a card, has no knowledge (of normal sort) of 
the disposition made of preceding cards; the specifications below will 
not apply without intelligent revision to fit the altered experiment. Many 
of the problems thus excluded are more entertaining, and undoubtedly 
better suited to illuminating the total “extra-sensory” question than is 
the simple card-sorting experiment here described. The controversies, 
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however, have centered around approximations to this experiment, and 
it is therefore the one with which we here deal. 


I. General Procedure. 

The method of simple matching involves: (1) thorough shuffling of 
the deck of cards, and (2) distribution of the cards to five stacks, each 
visibly marked with one of the five symbols borne by the cards. This 
distribution is made with attention to the possibility of assigning the 
cards to the stacks corresponding to the symbols borne by the several 
cards, and, presumably, with the desire to place each card in the “right” 
place. 

In casual tests, the subject is prevented from seeing the faces of 
the cards during the shuffle and “deal,” but is permitted to see the 
backs of the cards, and may even handle the cards, himself shuffling 
and dealing, and may record his own scores. 

Such conditions are not satisfactory in an experimental test. The 
backs of the ESP cards offer both visual and tactual cues to the face 
designs. Although, theoretically, these cues could be obviated by a care- 
fully planned lighting system and by requiring a certain technique in 
handling the cards, other forms of safeguard are more certain, and 
less productive of doubts on the parts of critics. 


II. Safeguards against Illicit Cues. 

1. The subject must not be permitted to see the cards, faces or 
backs at any time during the tests. This period of concealment, as a 
minimum, covers the time from the beginning of a shuffle to the com- 
pletion of the record of the distribution. The soundest method of as- 
suring the fulfillment of this condition is to provide, by proper screen- 
ing, that the cards employed in the test shall at no time be seen by 
the subject. 

2. The cards must not at any time be handled by the subject. All 
shuffling and dealing is to be done by the assistant, or assistants, or 
by one of the supervisors of the experiment. This restriction, and the 
one preceding, might be considered needless if ordinary playing cards 
of good manufacture were employed, but are indispensable with the 
ESP cards, and with various other cards which might be substituted. 

3. During a test, the faces of the cards must not be visible to an 
assistant or to any other person, and due precautions must be taken, 
by lighting arrangements and otherwise, to assure the non-reception 
of cues by the assistant. 














32 The JOURNAL of PARAPSYCHOLOGY 


4. No information shall be given to any subject concerning his suc- 
cesses, Or his scores, in the experiment, until after the termination of 
the service as subject. 


III. Distribution and Indications. 


1. Indicators of the five positions in which the cards are to be 
stacked will be cards of the same sort, from another pack. These in- 
dicators are to be plainly visible to the subject, although the stacks 
formed in the positions indicated are not to be visible. 

2. The order of the stacks must be the same in all experiments 
with all subjects. For example, the order selected, from left to right, 
may be: circle, cross, square, star, waves. This order then would be 
observed throughout the entire experiment. 

3. The subject may indicate the position in which each card is to 
be placed, by pointing to the indicator card, or by naming the symbol. 
Each subject may be allowed to choose the method (one of the two 
described) which he prefers, and shall then be required to adhere to 
this method throughout the total series of tests with him. The final 
report must contain notations of the method used by each subject, and 
certification that each subject was held to the method noted throughout 
the experiment. 


4. The cards are to be dealt (distributed) by the assistant, in con- 
formity with the foregoing specifications, to the stacks to which they 


are assigned by the subject. After laying a card on the assigned stack, - 


(in the assigned position), the assistant shall pick up the next card 
from the deck, and by a single word (e.g., “next”) shall signal to the 
subject to elicit the next assignment. 


IV. Scoring and Records. 


Immediately after the completion of the distribution of the cards 
in any “run of the deck,” the assistant shall turn the stacks over and 
shall record in ink the number of cards in each stack, and the number 
of “hits” (i.e., cards whose symbols correspond to the symbol of the 
stack) in each stack. These numbers are to be entered opposite the 
serial number for the test to which they belong, in a book ruled for 
the purpose of minimizing the probability of miss-entry. The entries 
for the several stacks must be in the same order, from left to right, 
as that of the stacks from the position of the assistant, and the symbol 
for each stack must be plainly stamped or drawn in the heading of the 
group of columns devoted to that stack. 
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Additional columns should be provided at the right of each page 
for the entry of total hits and misses for each run of the deck, and for 
such other entries as may be desirable. (See Section XI.) 

No more detailed scoring of hits and misses than is above specified 
should be undertaken, and no computations should be made during the 
experimental period. 


V. Quantitative Considerations. ; 

1. For initial purposes twenty or twenty-five subjects may consti- 
tute a working group. On completion of work with the first group, 
a second group may be started, and further groups employed if and as 
the data accumulated indicates the need of additional numbers. This 
method is deemed better than that of starting a large group of subjects 
at one time. 

2. The prescription of the minimal number of runs to be obtained 
from each subject offers some difficulty for two reasons. First, because 
the determination of sufficiency of runs can best be made from analysis 
of the actual data in the light of theoretical considerations. Second, on 
account of the unavoidable dropping out of subjects in a long experi- 
ment, the demand for sufficiency of number of runs per subject may 
conflict with the demand for sufficiency of number of subjects. In a 
critical experiment the number of runs should be high, in order that 
intercomparisons of fractions of the total series may be adequate, and 
that intercomparisons may be made with fractions of “pure chance” 
series. While 1000 runs per subject may be found necessary, it is 
possible that a smaller number may be acceptable. On the other hand, 
the series may need extension beyond 1000 runs. The number of runs 
must be the same for all subjects, as specified in Paragraph 3, which 
follows. 

3. Each subject must be put through the same number of tests 
(runs) as every other subject, in so far as this is possible; and the 
full records of each subject must be kept, regardless of the nature of 
the scores. If a subject unavoidably drops out of the experiment be- 
fore its completion, his scores, up to the time of his dropping out, 
must be preserved and reported along with the scores of the other sub- 
jects. In no case may a subject be given additional tests beyond the 
assigned number for the experiment. These conditions are vital for 
the experiment, and must be observed if statistical measures are to be 
applied to the data resulting. 

4. Working conditions must be as uniform as possible. The length 
of work periods must be the same, and the number of tests per work 
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period must be the same, for all subjects. Since different subjects 
will have different rates of work, the distribution of tests through the 
work period should be on the familiar plan commonly known in lab- 
oratories as “headway” (1.e., ten minutes headway, five minutes head- 
way, etc.). 

The distribution of work periods through the week also should be 
as nearly uniform as possible. 


VI. Selection of Subjects. 

As a precaution, it would be desirable to pick all subjects from a 
group having a relatively short range of ages and an approximation to 
uniformity of educational status. This condition may be met, for ex- 
ample, by picking from two adjacent college years, as: sophomore and 
junior. 

Since the question of sex differences may eventually rise in con- 
nection with any experiments in this field, it will enhance the value 
of the data if the subjects are either all of the same sex, or else divided 
nearly equally between the two sexes. 

As regards sophistication, there are a number of possible selective 
procedures, of which two are important. On the one hand, we might 
give preliminary tests of “ESP” to a large number of persons, and 
take as subjects for the experiment only those who showed a “positive” 
score in the preliminary tests. This method of selection would perhaps 
appeal to the person already convinced of the truth of the ESP hy- 
pothesis, and would not, for the sceptic, invalidate the experiment, 
provided full details of the method of selection were presented along 
with the report on the experiment. On the other hand, one might 
select as subjects only persons who have had no tests of any sort bear- 
ing on the question of “ESP” or other occult phenomena, and on whom 
therefore no preliminary ratings could be made. This method of se- 
lection would undoubtedly appeal most strongly to many experimental 
psychologists who are interested in all aspects of the card-sorting opera- 
tion. 

One or the other of these two principles of selection should be 
adopted in preference to any intermediate principle. 


There is no objection to the use of paid subjects, provided all sub- 
jects distinctly understand that they are to be employed for the full 
time of the experiment regardless of their scores; and that any further 
employment, in similar work, after the completion of the experiment 
outlined, will be without pay. 
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VII. Orientation of Subjects. 


Every subject should be shown a sample deck of “ESP” cards, 
and be informed briefly of the purpose of the experiment namely, that 
it is to find if, or to what extent, a subject who desires, or intends, to 
distribute the cards in accordance with the face-symbols, but without 
any sensory indication of the symbols on particular cards, can succeed 
in such distribution. No casual trials with a deck should be permitted ; 
but further instruction must be given in the first actual test, the re- 
sults of which must be regularly recorded. 

Each subject shall be requested, as a matter of honest codperation, 
to avoid, during the continuance of the experiment, handling or in any 
way using any deck of “ESP” cards, or of similar devices, except in the 
experiment itself. 


VIII. Selection of Assistants. 


The assistants must be persons who have had adequate training in 
experimental psychology. A graduate student who should have had at 
least one year of laboratory training, involving practice in methods, 
techniques, and precautions in a variety of experimental topics; and 
who has conducted individual research of laboratory character for at 
least a semester; would have the minimal acceptable qualification in 
respect to preparation. The qualification of known reliability, and care- 
fulness in work, is especially important. 

No person whose stipend or position depends in any way on the 
outcome of the experiments shall be employed as assistant. 


IX. Superintendence. 


The experiment should, throughout, be under the direction and con- 
trol of two or more psychologists who are regarded by members of 
the profession generally as competent in the experimental field. One 
of these superintendents must be on duty during every work period, 
and have actual oversight of the conduct of the tests. 

In view of the present situation, and the need of a definitive ex- 
periment, it is highly desirable that the experiment be set up under 
the superintendence of three psychologists, each from a different uni- 
versity. 


X. Protection of Materials and Records. 


All cards and other materials, including all records, when not in 
actual use, should be deposited in safe keeping, preferably in a safe, 
in such place and manner as the superintendents may direct. At the 
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end of each work period, the records for that period must be turned 
over to the superintendent on duty, who shall be responsible for such 
records. 


XI. Elaboration of Data. 


Although the details of tabulation and other elaboration of data 
cannot be prescribed in advance for a comprehensive experiment, there 
are certain points which have been indicated by past reports as of signal 
importance. 

1. The scores for each subject should be totaled (A) for each set 
of five successive tests (runs of the deck) ; (B) for each successive set 
of twenty-five runs; (C) for all the tests of each work period; and 
(D) for the total tests of the experiment. This tabulation, in each 
case, should give (a) the total of hits and the total of misses for each 
of the five stacks separately, and (b) the totals for all five stacks to- 
gether. 

2. Based on the tabulated data above indicated, the average success 
of each subject, for the several groups of tests, should be computed 
in two ways. (A) Average hits per test; computed by dividing the 
total number of hits by the number of tests. (B) Ratio of success (or 
percentage of success) ; computed by dividing the total number of hits 
by the total number of cards. 

3. Tabular or graphic presentation must be made of the distribution 
of scores for the whole group of subjects. This may be in terms of 
total hits; average hits per test, or ratio of success. This presentation 
must bring out clearly the critical point, which has been involved ex- 
plicitly or implicitly in almost all criticisms of ESP tests, of the relation 
of positive scores to negative scores and to neutral scores. 

The presentation should make evident, in particular, evidence as to 
the occurrence, in the group, of subjects falling in any or all of the three 
following classes: (a) Those having a success ratio (or hit average) 
for the total experiment, or a substantial part thereof, above the ratio 
which may be accepted as a typical “chance ratio” or “chance average.” 
(b) Those having success ratios differing negligibly from the “chance 
ratio.” (c) Those having success ratios distinctly below the “chance 
ratio.”” Changes of subjects, from time to time during the experiment, 
from one of these classes, to another, if such changes occur, should be 
made clearly evident. 

4. The determination of the “chance average” and of significant 
deviations from this average present problems which have become con- 
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troversial. In a strongly skewed distribution, the relation of arithmetic 
mean, mode, and median is always a matter of interest; and in regard 
to the distribution of ESP cards there seem to be differences of opinion 
concerning the relation of the most probable score for a single run of 
the deck to the measures of distribution of a large number of scores. 
Although it seems hardly possible that these considerations can be de- 
cisive in the interpretation of ESP scores, nevertheless, an experiment 
carried to adequate length by the procedure outlined in the preceding 
sections may throw light on a problem which, apparently, is not to be 
solved by mere mathematical postulation. 
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COVARIANCE METHODS IN THE COMPARISON 
OF EXTRA-SENSORY TASKS 


GARDNER MURPHY and ERNEST TAVES 


Columbia University 





ABSTRACT: Our hypothesis is that if extra-sensory perception is present in 
a long series of calls, to a degree discoverable by statistical methods, it should be 
revealed in a comparison of various parts of the data with one another. Thus, if 
conditions are favorable to ESP in general, high scores on one task should be ac- 
companied by high scores on another task; and if conditions are so unfavorable as 
to produce depression of scores below chance expectation, the low scores on one 
task should be accompanied by low scores on other tasks. 

Four tasks were used, in which thoroughly shuffled cards in a cubicle at Co- 
lumbia University were to be “called” by the subject: (1) Playing-cards, in which 
odds are 1 in 52 for direct hit and 1 in 2 for color (red or black), (2) ESP cards, 
in which odds are 1 in 5, (3) circles and crosses, in which odds are 1 in 2, and (4) 
circles and blanks, in which odds are likewise 1 in 2. 


Each experimental unit is the completion of this “quadruple-task” in the order 
listed. Scores in one task at a given sitting are to be compared with scores in the 
other tasks obtained at the same sitting. The present report includes data from 
1000 such experimental units of four tasks each—1000 quadruple-tasks (QTs). 


The subjects in the experiment were 74 persons, most of the data coming from 
five major subjects, two of whom were the present writers. About half the work 
was done with subjects a few feet from the cards, which were on the other side of 
a celotex wall; the other half was done at distances of from a quarter-mile 
upwards. 

The results indicate, in general accordance with the hypothesis, that high scores 
on color and suit in the playing-card task are accompanied by above-chance scores 
on the circle-cross and circle-blank tasks. The obtained difference between hits on 
the circle-cross and circle-blank tasks when color scores are high and when color 
scores are low is of a magnitude which should occur through the operation of 
chance alone about three times in a thousand. 

Three of the five major subjects show the covariance of color scores and circle- 
cross, circle-blank scores to a markedly suggestive degree. Nevertheless, there are 
several phases of the distributions of the data which are difficult to interpret, and 
it is considered wise to continue the experiment, using new subjects as well as con- 
tinuing with the old. There is suggestive evidence that awareness of one’s own 
trends can disturb the covariance effect; for this reason those repeating the exper- 
iment are urged to avoid giving any suggestion to subjects regarding results 
obtained elsewhere. 





INTRODUCTION 


‘i PRESENT study is primarily an exploratory analysis of the pos- 
sibilities of a method. When the ultimate nature of a phenomenon is 
unknown, or its occurrence not predictable, it may be impossible to state 
it as a function of a specific variable, and yet the phenomenon may ren- 
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der itself evident in two or more ways at once, so that one manifestation 
of the phenomenon will accompany another. Psychologists have long 
recognized through the use of the coefficient of correlation the need to 
study the interdependence of phenomena, even when the conditions 
determining the phenomena remain to be discovered. 

Our hypothesis is that if very slight traces of extra-sensory phe- 
nomena are present in a series of calls, they may be discovered by apply- 
ing standard statistical methods to various parts of the data, to see 
whether such parts of the data as should be uncorrelated on a chance 
basis will show significant quantitative interrelations. We refer here to 
various types of measures such as are obtained by the application of the 
product-moment, contingency, and Chi-square methods. 

Our procedure, then, was to study the efforts of subjects to call 
various types of concealed material at a single sitting, and to ascertain 
whether correct calls on one task at a given sitting are significantly re- 
lated to correct calls on another task at the same sitting. If there is any- 
thing about the sitting which is favorable to ESP, the scores on the 
various tasks might be positively related. 

Whereas most investigations in this field have been concerned with 
exploring the powers of unusually gifted subjects, our interest was to 
see what would happen when very large quantities of material were col- 
lected from ordinary persons under reasonably uniform experimental 
conditions. The material to be guessed by the subjects was to be 
definitely beyond range of all the senses, and the statistical method was 
to be a simple standard procedure, the applicability of which is uni- 
formly recognized. The experiment has been in progress only about a 
year and a half and is as yet very far from complete. It is hoped that 
publication at this time will result in a variety of comments which will 
make possible clarification of issues and improvement of techniques and, 
above all, make possible the repetition of the procedure by any psychol- 
ogist who has the time and interest, using groups of ordinary persons 
who have the time for sustained inquiries into the problems at hand. 


LABORATORY AND GENERAL CONDITIONS 


In a corner of GM’s office at Columbia we built a 5’ x 6’ celotex 
cubicle, with a roof 8 feet from the floor, as shown in Figure 1. The 
position of the one door of the cubicle is indicated in Figure 2. This 
door was always shut during experimentation; the one window of the 
cubicle was always covered with a tight cardboard screen. Within the 
cubicle was a card table, upon which the cards to be guessed were placed 
by the experimenter, no other person being present in the office while 
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the cards were shuffled and arranged ; the door of the office (D in Figure 
2) was locked during this time. Subjects for the experiment were ad- 
mitted to the office to make their calls, but of course were not admitted 
to the cubicle. GM and ET both served as experimenters. 
























































Peer 
Cubicle —— 
Card table a | pews oak 
DesR 
Desk 
Da 
Hall 





Ficure 2. FLooR PLAN OF OFFICE. 


MATERIALS AND PROCEDURE 


After considerable exploratory experimentation with the Duke Uni- 
versity ESP cards (star, circle, cross, square, waves) we were im- 
pressed by the suggestion of a physicist that it might be wise to work 
with several different types of material, in order to make possible a 
comparison of results. It was suggested that in terms of the general 
psychology of perception it should be easier to distinguish between two 
symbols than between five, and still easier to distinguish between the 
presence of a symbol and the absence of that symbol. We therefore de- 
cided to set up a uniform experimental procedure in which three tasks 
were to be completed at a single sitting. The first of these tasks con- 
sisted of 25 of the ESP cards, to be guessed by the down-through 
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PHOTOGRAPH SHOWING OUTSIDE OF CUBICLE AS SEEN FROM 
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method.! The second task was to call, down-through, a deck of 25 cards, 
each of which bore either a circle or a cross. In the third task, which 
also consisted of 25 cards, each card either bore a circle or was entirely 
blank. 

Exploratory work with the “triple-task,” as we called this procedure, 
made evident the necessity for a number of improvements: (1) The 
addition of playing-cards, in which many subjects were interested. The 
long odds in calling a playing-card exactly right, being 1 in 52, made the 
task interesting. Also, most subjects are familiar with playing-cards and 
like to guess them. (2) We felt it advisable to have two exactly com- 
parable tasks of each type used. Consequently we made it a rule to use, 
for each of the three tasks, a deck of 50 cards broken into two series 
of 25 each. 

In working with the triple-task it seemed to us desirable to prevent 
the subject from working by a supposed process of elimination, which 
would be expressed by saying, “I have called a lot of circles; I’d better 
not call any more of them.” Although such a procedure on the part of 
the subject does not change the mean chance expectation, it does affect 
the standard deviation of the distribution of scores. This problem was 
partly solved in the work with the triple-task. In the case of the ESP 
cards, for example, 10 of each symbol were shuffled together, making a 
deck of 50. The deck of 50 was then dealt out, as in dealing to two 
persons, into two cardboard boxes, just large enough to contain the 
deck. Thus the distribution of symbols in a deck of 25 was not limited 
to 5 of each, with the result that the subject felt less compulsion to 
“even-up” his calls. The circle-cross and circle-blank decks were pre- 
pared in the same manner. This method of preparing decks to be called 
does not give an absolutely binomial distribution, but the sigma correc- 
tion is slight and is easily applied.* 

One further change in the preparation of material must be noted. 
Subjects found it difficult to call as many as 52 playing-cards in order, 
and for this reason the number to be guessed was cut to 25. This deck 
of 25 cards was always the top 25 cards from a shuffled deck of 52. 
Also, some subjects preferred to have these cards spread out on a table 

*In the down-through (DT) method the shuffled pack of cards is to be guessed 
one at a time, from top card through to bottom card. 

* At a later stage in the experiment, subsequent to collection of most of the 
present data, we arranged to have on hand 500 each of the ESP, circle-cross, and 
circle-blank cards. From each such source of 500 cards, 50 were chosen at random 
for a given experiment. On this basis, the maximum sigma is 1.001 times the 
theoretical sigma, and corrections may be disregarded. (Greenwood, J. A., and 


Stuart, C. E., Mathematical techniques used in ESP research, Jour. Parapsychol., 
1937, I, 206-225). 
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(always within the cubicle and out of sight, as described earlier) rather 
than arranged for the down-through (DT) procedure; we fell into this 
practice until we had gotten so much data in hand as to be reluctant to 
change it. The scores obtained on playing-cards, as will be seen later, 
are nevertheless comparable with those on which the DT procedure was 
used. (Since under these conditions the target card cannot appear twice, 
there is a slight sigma correction to be made if direct hits appear 
markedly high or low). 

The set-up just described, involving the four tasks, (1) 25 playing- 
cards, (2) 50 ESP cards, (3) 50 circle-cross cards, and (4) 50 circle- 
blank cards, was finally determined upon on December 4, 1937, and has 
continued to the present writing. The present report covers only the 
quadruple-task method, with the tasks as finally specified. 

The method of administering the quadruple-task, or QT, may now 
be defined in detail. We shall present first the original procedure, as of 
December, 1937, and then discuss several minor improvements upon it 
which were made during the course of the present work. 

The experimenter, alone in the office, with the door locked, prepared 
the experimental targets by the following procedure: He entered the 
cubicle, dovetail-shuffled® the entire deck of 52 playing-cards five times, 
and dealt out, from the top of the deck, 5 rows of 5 playing cards each, 
face down, on the card table, as shown in Figure 3. The order in deal- 
ing, as in calling the cards, was always left to right, top to bottom, as 
in reading a book. He then took the 50 ESP cards, put them through 5 
dovetail shufflings, and dealt them out between two cardboard boxes, 
put the lids of the boxes in place, and placed the boxes in the southwest 
corner of the table. The procedure was the same with the 50 circle-cross 
cards, which were placed in the middle two cardboard boxes shown in 
the photograph. The experimenter then did the same with the circle- 
blank cards, which were placed at the southeast corner of the table. The 
disposition of the entire set-up is shown in Figure 3. 

The experimenter then came out of the cubicle into the office, closing 
the cubicle door, went to the outer door of the office, and admitted the 
subject or subjects who wished to participate. In the experimenter’s 
presence the subject took a record sheet* and indicated upon it, in order, 
his guesses as to the 25 playing-cards, the 50 ESP cards, the 50 circle- 
cross, and the 50 circle-blank cards. The experimenter then went into 
the cubicle, brought out the target cards, made the record of the target 


* The pack of cards, face down, was cut by hand, and the two parts of the pack 
allowed to dovetail together, being then squared off by hand. The last shuffling 
was followed by an additional cut. 

*A record sheet is shown in Figure 3a. 
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cards upon one or more of the call sheets, and scored the four tasks. The 
playing-card task was scored for color, suit, number and direct hit. The 
remaining three tasks afforded but one way of scoring—for direct hits, 
or correspondences, only. 
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Beginning early in February, 1938, it was made an invariable prac- 
tice to shield the call order while recording the target order. Beginning 
on October 5, 1938, the experimenter recorded the target order on an 
independent sheet before looking at call orders. 

In addition to the method just described, we used a long-distance 
procedure, the subject making his calls wherever he happened to be, and 
whenever he cared to. This method, of course, required that the time of 
making the calls be precisely indicated on the call sheet. It was also 
necessary to establish a definite routine for preparing the target cards 
regularly each day. The times decided upon were 10 A.M. and 5 P.M. 
If the subject’s call sheet indicated that the calls had been made while 
the target was being changed, the sheet was, of course, discarded. 

Material of this long-distance type formed a very small portion of 
the total data collected up to June, 1938. But having worked mostly at 
short distances during the academic year 1937-8, we continued the QT 
method during the summer of 1938 on a long-distance basis. ET was in 
Denver, Colorado and in Richmond, Virginia for various parts of the 
summer; GM was in Berkeley, California. One other subject, RF, was 
in New York part of the time, travelling through various parts of the 
country the rest of the time. Keeping a record of the time, she called 
both the Columbia target and a similar target prepared in Berkeley by 
GM. Almost all the summer work is from these three subjects. 

The California target was prepared by GM according to the method 
described for the New York target, and was kept in his office (which 
was locked except when he was there) at the University of California, a 
large drawer being used to spread out the cards (in the absence of a 
cubicle.) During the summer the New York target was prepared twice 
daily in the cubicle by GG, a graduate student, who regularly sent dupli- 
cate records of the New York target orders to GM at Berkeley. ET 
sent his calls to GM, as did RF. We resumed the QT work at Columbia 
in September, 1938, having at that time 597 QTs on hand; at that time 
we determined to complete the present series, working up the data when 
the number of QTs reached one thousand. 


DEGREE OF ADEQUACY OF SHUFFLING 


Despite the rather elaborate shuffling procedure, a test of the ade- 
quacy of the shuffling was made. The relative frequency with which 
each symbol appeared at various points in the deck was computed. 
Tables I and II give such distributions for two sets of 50 targets each, 
one set having been prepared by GG, the other by ET. It will be noted 
that the discrepancies from chance expectation are non-significant. Sec- 
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ond, the possibility that imperfect shuffling leads to some connection be- 
tween successive targets was investigated. Targets were checked against 
succeeding targets, and correspondences were noted. Table III presents 
the results of the test, showing no significant relations between successive 
card orders. 

An additional control is offered to test the existence of a possible 
linkage between different parts of the same target, e.g., whether there is 
any extra-chance relation between color of playing-cards at the various 
points in the target and the arrangement of circles and blanks in the 
same target. Table IV suggests no relation of this sort. 


TABLE I 


RESULTS OF EMPIRICAL TEST OF SHUFFLING, BASED UPON 50 TARGETS SHUFFLED BY G6, 

SUMMER OF 1938. THE TABLE SHOWS DEVIATIONS FROM EXPECTED FREQUENCY FOR 

VARIOUS SYMBOLS FOR THE FIRST FIVE AND LAST FIVE POSITIONS IN THE DECK. TWO 
SIGMA VALUES ARE GIVEN AT THE FOOT FOR COMPARISON. 
















































































TASK 
Position | Playing- 
a. cards ESP deck 1 A ESP deck 2 0+1 042 O—1 O—2 
lec 
Black o oO + Ojfjsi« oO + oj;=—|0 oO Oo 1@) 
1 —l —) 1 0 6 | —6 2)-—1 1 0} —2);—1|—9 6 4 
2 —4 0}; —2 1|—4 5 |—2 5 o|;—2/];—1]|—4 §;—1 2 
3 —6 —2;—1 0 1 2 1|—3 0 5|—3 2 2|—7 2 
4 1 —l 5|—4]|]—3 3};—1}|—3 0} —1 5 7 + 1 0 
5 —4 3};—1|-—3}—1 2|;-1 0 3 2)|—4|—6/]—2/|—4/—-—8 
21 oan —4 | —2 | —2 4 4)—2/)-1 7|—4 0;—1;—5 0 3 
22 2 3 0 2}—1 | —4|—1 | —2 | —2 0 5 2 3|—2 3 
23 0 —2 0 2|-—3 3|—3 0 0;—1 4 6|—3 1 4 
24 —1 0 | —2 1 | —3 4 1 | —2 1; —1 1 0 2\|—7 5 
25 —2 2 1 2);—2/)]-—3 0 4 o}—1|}—3|—5 | —3 | —8 | —2 
20 7 my 7 
TABLE II 
SAME AS TABLE I, BASED UPON 50 TARGETS SHUFFLED BY ET, OCTOBER, 1938. 
TASK 
Position | Playing- 
a. cards ESP deck 1 ESP deck 2 0+10+2 O—1 O— 2 
ec 
Black * + 0 oO —\ir* + 0D oO =—|o0 1) oO oO 
1 4 2 0 1 | —2|—-1 0 4|-—1|—4 1 0 0; —1 | —2 
2 =] —l1 | —4 3 3|;—l 4};-—1/|—1 4)—6 0 5|—1 0 
3 0 —1 |} —1 3|}—1 0} —6 1 0 1 4 3 4|—7 1 
4 —4 6 0 | —2 o|—4 2};—1 1 1|—3 |} —2 6 12 1 
5 5 —3 2 §|}—2|-—2)}—1 0 | —2 1 2;—5 q 0;—1 
21 8 1 0; —1 5|—5 0 1 3|;—4 vu | —3 4|;—6/]—1 
22 3 —2 0} —2 6 | —2 2 2 1|—3 | —2 5|—4|—4|—4 
23 —2 -—4 2 0 3|;—1 5|—4}-—1 1} —1}]—1 | —2 1 0 
24 —1 2} —3 41—3 0;—1/]—1 1 5|—4 0;-—1 5|—3 
25 3 5 4 3 | —2 0 2|—3 3}—1]|—1 5 3 1 4 
2c 7 5.7 7 
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TABLE III 
RESULTS OBTAINED COMPARING ONE HUNDRED TARGETS AGAINST SUCCEEDING TARGET 




















Correspohdences | Correspondences 
Task expected obtained D/*D 
ctagiomcande (color) (25)...... 1250 1261 0.44 
oS nsseenee 1000 1026 0.92 
Circle-cross (50).............. 2500 2553 1.50 
Circle-blank RR ripe 2500 2492 0.25 
Combined 0+, O— (100)...... 5000 5045 0.90 
TABLE IV 


TABLE SHOWING AMOUNT OF LINKAGE BETWEEN PLAYING-CARDS AND THE SECOND } 

CIRCLE-BLANK DECK. THE TABLE SHOWS THE DEGREE TO WHICH A RED (OR BLACK) 

PLAYING-CARD AT ANY POSITION IN THE PLAYING-CARD DECK TENDS TO BE ACCOM- 

PANIED BY A CIRCLE (OR BLANK) IN THE CORRESPONDING POSITION OF THE CIRCLE- 
BLANK DECK. BASED ON 100 TARGETS. 





























Piayinc Carps 
. Red Black Total 
Circle-blank 2 (circle)... ..... 653 617 1270 
Circle-blank 2 (blank)........ 618 612 1230 
EG EE ear! 1271 1229 2500 
CRORE. 2. obo de cs ccod 
pte Ree eee ree 56 


Evidently no significant linkage is shown here. Further data on 
linkage appear below, pp. 62-65. 


METHOpD oF ScorING® 


The method of scoring when GM or ET was testing an individual 
subject during the period to the beginning of February, 1938, was to 
write the order of target cards in a column parallel to the record in 
which the subject’s calls had been entered. But Kennedy® has shown 
that a graduate student opposed to ESP, recording by this method, 
made 9 errors in 2625 calls of the ESP cards, of which 1 increased the 
score, 3 decreased it, and 5 left it unaffected. It is possible that there are 


5 The question of adequacy of scoring cannot, we believe, be completely dis- 
cussed to the satisfaction of all by any method in which cards are recorded by hand 
—one reason for the development of the machine-photographic procedure, de- 
scribed in another article (p. 11) in this issue of the JourNAL. Nevertheless we 
wish to report as fully as we can on the problem of accuracy in recording. 

- aun J. L., in The ESP symposium at the APA, Jour. Parapsychol., 1938, 
, 247-272. 
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considerable individual differences in the magnitude of such errors and 
in their direction. It will be noted later that the ESP curves, as well as 
the others, show no traceable effects of such a constant error, but this 
point is not a complete answer to Kennedy’s question regarding record- 
ing errors. 

During the latter part of the academic year, 1937-8, another method 
was used in the case of group experimentation, and this throws light 
upon the general question of accuracy of recording. When several 
subjects competively called the same target, both experimenters being 
present, one of them acting as subject, the other having prepared the 
target, the sheets were handed to the subjects to give them the satisfac- 
tion of immediate checking, while the experimenter made his own check- 
up, observing one of the subjects, and in the immediate presence of the 
others. As he turned over the cards and recorded them, he called out the 
order and scored one call sheet, while other subjects scored one another’s 
papers. This, of course, was not “official” scoring; the experimenter 
later checked all such records, comparing them with his own target 
order. The number of discrepancies turned out to be approximately 
.001 of the total recordings. The figure just given indicates only the 
discrepancies found between the target order as recorded by the experi- 
menter and the target order as recorded by subjects, making their rec- 
ords as the experimenter called out the order (it was noted above that 
shortly after the experiment got under way we began to shield the rec- 
ord of call orders when entering target orders). 

Prior to the beginning of the Summer distance work, all data sheets 
were checked by the experimenters at least twice after the original 
scoring. Errors, as they were found, were rectified, but no record was 
made as to the exact number of errors found. All the data from this 
point on, however, were subjected to a rigid checking by various grad- 
uate students, under pressure to find as many errors as possible. 582 
data sheets were re-checked on this basis, and errors as shown in Table 
V were found and corrected. The 418 QTs which were obtained prior 
to the beginning of the Summer work had been as rigidly checked, but, 
as stated, a record of errors found was not made. 

On October 5, 1938, at which point 597 QTs had been gathered, the 
independent target record method was adopted, on the basis of sugges- 
tions made by Coover and Kennedy. Since the experimenter was, of 
course, never allowed to call a QT which he had prepared, the making 
of the independent target record was an integral part of the procedure 
in all QTs thereafter. Target orders and call orders were regularly 
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juxtaposed for scoring; while shielding the subject’s calls, the target 
was recorded in columns adjacent to those in which the subject had 
recorded his guesses, and correspondences were then noted. Since June 
6, 1938, target records have been made in duplicate. We do not know 
how autistic factors may work, and we wish to emphasize the correct- 
ness of Kennedy’s view that it is wise to make an independent target 
record. 








TABLE V 
MAGNITUDE OF ERRORS FOUND IN CHECKING, SHOWING DIRECTION OF DISCREPANCY 
(582 gts). 

Task Errors Positive Negative % 
ican Te aI Ad 15 1 14 .001 
Mec a oh), 5.04 amie asken 4 1 3 .0003 
BS URS ipgeare care 26 1 25 .002 
I 6 ee 2 F 0 .0001 

erg pl aealeaiper igs ae 18 2 16 .001 
BE a ee 18 0 18 .001 
ee F 0 7 .0005 
Circle-cross 2................ 4 2 2 -0003 
Civele-blank 1................ 6 1 5 .0004 
Circle-blank 2................ 4 1 3 .0003 

















Although nearly all the data were collected under our eyes, or under 
long-distance conditions, permitting our scoring of the call sheets before 
anyone else had access to them, we are entirely willing to conceive of two 
possible hypotheses: (1) Either or both of us is so emotional or so dis- 
sociated as to have falsified the data. (2) Someone gained access to our 
files and tampered with the record sheets. 


Just such hypotheses as these necessitate repetition of the exper- 
iment, not once, but many times by independent groups of experimenters. 


SuBJECTS 


Our subjects were all those who were interested and able to give the 
time. No one who volunteered for the experiment was ever refused, and 
no one was ever discouraged from taking further tests. We did, of 
course, make it a practice to try subjects who were interested in the 
problem, but we gladly included persons of every attitude and per- 
suasion. The present study made use of 74 subjects, the number of 
times each subject went through the experiment depending primarily on 
his or her own interest. These subjects were largely undergraduate and 
graduate students of both sexes, and mostly under 30 years of age. 


We shall arbitrarily use the term major subject to include those who 
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did more than 60 QTs each. The largest number of QTs called by any 


‘minor subject was 52. These five persons were the present writers, a 


woman graduate student at Teachers College, Columbia, a woman under- 
graduate student at Barnard College, Columbia, and a woman who had 
formerly been a graduate student at Duke University, and had worked 
in parapsychology there. 

_ Two groups of minor subjects may be more specifically described: 
(1) The Executive Secretary of the American Society for Psychical 
Research asked us in June, 1938, whether we would be willing to carry 
out some experiments with members of the society at the society rooms, 
located at 40 East 34th Street, New York City. We replied that we 
should be glad to carry out such experiments provided we could proceed 
in the same way in which we worked with college students, that is, 
under strict conditions and requiring a great deal of arduous work from 
each subject. The plan was approved by the trustees of the society, a 
small fund for apparatus and two assistants were provided, and work 
started in October, 1938, and continues at this writing. 

The only phase of this work which is pertinent in the present connec- 
tion is the QT calling which has been done by the members of the ASPR 
experimental group. Several members of the group came by appoint- 
ment to the laboratory to do a few QTs under the regular routine condi- 
tions (sitting at desks in the laboratory or in adjoining rooms, with the 
cards in the cubicle, as described previously). Most of the members of 
the group have never seen the laboratory. Twice a week since October, 
experiments at the rooms of the society have included long-distance 
calling of the Columbia QT under the same conditions prevailing in 
other long-distance work and, in addition, four of the subjects have 
turned in regularly, at the experimental meetings, distance QTs which 
they had done in the time intervening between group experiments. 
These records received by GM and ET were scored against the pre- 
viously recorded targets. We included these in the totals merely because 
it is uniform practice to include all QTs from all persons participating. 
The ASPR material began to come in so late that in the first thousand 
QTs there are only 52 from this source. 

(2) On November 13, 1938, GM gave a short talk on ESP methods 
to about 100 students at Bennington College, Bennington, Vermont, de- 
scribing the covariance method, and graphing the results from the first 
817 OTs. He asked for cooperation from students in long-distance call- 
ing, which was to be under the direction of Dr. and Mrs. Dwight Chap- 
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man, of Bennington College. The students were to make their calls at 
convenient times in the day, to hand these to the Chapmans, and to have 
them forwarded to Columbia for checking. A number of students re- 
sponded, and are continuing to call the QT at this writing. 


RESULTS 

The first and most obvious way to treat the data is in terms of the 
distribution of correspondence as compared with those expected by 
chance. These distributions appear in Figures 4 to 11. It is evident 
that the mean deviations from chance expectation on the various tasks 
are small and non-significant. Also, the data for the individual major 
subjects are fair approximations to the expected chance frequencies (all 
obtained scores are presented in Appendix I, arranged, by subjects, 
chronologically ). 
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In spite of the apparently non-significant distributions of scores, 
various interesting trends in the data began nevertheless to appear 
during the spring and summer and could not escape attention. Most 
striking was the tendency for high scores on the color or suit of the 
playing-cards to be accompanied by high scores on the circle-cross or 
the circle-blank task, or the sum of the two, although no association with 
ESP scores was apparent. Whatever it is that makes the person score 
high in playing-cards is for some reason correlated with a tendency to 
score high on circle-cross and circle-blank. A similar linkage appeared 
when the playing-card scores were low; a low score on color and suit 
in the playing-cards proved to be correlated with low scores on the 
circle-cross and circle-blank tasks. There was no reason, as far as we 
could see, why this should be the case if we were dealing with the 
operation of “chance,” and the matter called for close study. 
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Ficure 5. DistriBUTION OF SUIT SCORES. 
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Ficure 7. DISTRIBUTION OF “DIRECT HIT’ SCORES. 








METHODS IN THE COMPARISON OF ExtraA-SENsORY TASKS 53 


=? 


Frequency 


Sern om vers seesea 


Frequency 


= % 





"eee 8 8588 SSP SReSE arr 






Mean 10.088 


%D 0.95 








+ g 
"| 
a 3 5 J ry rT 13 16 17 19 at 
Score 


120 


@ 
(=) 


n 
o 


~ 
oS 


20 


Ficure 8. Distrisution or ESP scores. (Unit or 50 cALts.) 


Ferra 6 
SGeennr*" 9 


109 
3124 
104 


Mean 25.079 


D 
ee 


25 











nu 12 #16 WW © ww 2 «625 27 Ul OS 
Score 


Ficure 9. DistriBuTION oF o + scores. (Unit or 50 CALLs.) 
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Figure 10. DistrisuTion oF 0 — scores. (UNitT oF 50 CALLS.) 
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Figure 11. DistripuTIon oF 0 +, 0 — scores. (Unit or 100 CALLS © 
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Figures 12 and 13 indicate the covariance situation as it stood at the 
end of the first thousand QTs. The method used in plotting the curves 
was as follows: The values of the constant score were marked on the 
abscissa. Then the averages of all variable scores obtained at the same 
time as the various constant scores were computed, and plotted on the 
ordinate. In Figure 12, for example, in which color is the constant and 
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Ficure 12. CURVE SHOWING RELATION BETWEEN COLOR AND 0 +, 0 — TOTAL 
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Figure 13. CURVE SHOWING RELATION BETWEEN COLOR AND 0 +, 0 — 
SCORES, 0 +, 0 — AS CRITERION. 











56 The JourNAL of PARAPSYCHOLOGY 


circle-cross, circle-blank combined total the variable, the first point on 
the curve indicates the average of all circle-cross, circle-blank scores 
obtained whenever color score was 5 or 6. The number of cases upon 
which each point is based is given in the N column above the curve. 

Figure 12, in which color is plotted on the abscissa, is regarded as 
slightly to be preferred to Figure 13 because the number of calls at each 
point is sufficiently large to permit a simple representation of the situa- 
tion with no grouping except at the extremes. Neither curve differs 
significantly from a straight line with a slight, but significantly positive, 
slope. 

This does not mean at all that we are dealing with a high correlation. 
The product-moment r for the entire thousand is .11 with a PE of .02 
and a P value, based upon ¢,’ of .003. The P based upon ¢r is .0002, 
but the more conservative value based upon ¢ is preferred. 

However meaningful or meaningless the mass trend, the individual 
trends need to be reported separately and appear in Table VI, which 


TABLE VI 


PRODUCT-MOMENT 7S AND ¢ AND P VALUES FOR FIVE MAJOR SUBJECTS. THE CORRELA- 
TIONS ARE BETWEEN SCORE ON COLOR AND SCORE ON CIRCLE-CROSS, CIRCLE-BLANK TOTAL, 








Subject n r t P 
See 140 oat 3.336 Less than .01 
ee 232 .14 2.144 Between .05 and .02 
I asain 69 .19 1.584 Between .10 and .20 
. eee 62 Re -936 Between .30 and .40 
ieee 102 .09 .904 Between .30 and .40 

















represents the data from the five major subjects. Figure 14 shows the 
scatter of the entire set of scores. 


FURTHER CORRELATIONS 


The intercorrelations of the four tasks were computed, with results 
as shown in Table VII. 

It is apparent that none of these correlations is significant. In par- 
ticular, circle-cross is not correlated with circle-blank. Though cor- 
relations of color with circle-cross and of color with circle-blank are 
positive, it is only when circle-cross and circle-blank are combined that 
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‘= Gan) in which nm is the number of comparisons and r is the coeffi- 
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TABLE VII 
TABLE SHOWING PRODUCT-MOMENT ?’S OBTAINED BETWEEN VARIOUS SCORES 
(N = 1000). 

Scores r r/or Pi* t Ps** 
Color ESP...... —.01 31 .38 316 7— .8 
Color O+....... .07 2.19 -02 2.212 .02— .05 
Color O—....... -05 1.56 .06 1.580 -l— .2 
Suit ESP...... —.01 ost .38 316 -7— .8 
moe Se. ...... .02 -63 -26 -632 5— .6 
Suit O—....... .04 1.25 ll 1.264 2— 3 
ESP O+....... .00 -00 -50 -00 1.00 
ESP O—....... —.02 -63 -26 -632 5— .6 
O+ O-....... —.01 oat .38 -316 7— .8 

*P,= P based upon r/ cr. ° 


**P= P based upon t. 


a clearly significant correlation with color is given. Whether this fact 
should logically suggest that the P value of .003 (in Table VIII) is an 
insufficient anti-chance value is partly a question for statistical judg- 
ment, but we think chiefly a point in favor of further research. If the 
value is an artifact, it will probably not reappear in a repetition of the 
experiment. 
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Ficure 14. ScCATTERGRAM OF COLOR AND 0 +, 0 — SCORES. 


It appears to be only when the QT is done at one sitting, that is, as 
part of one psychological task, that the scores show the relation por- 
trayed in Figures 12 and 13. In the same connection it is doubtful 
whether there is any carry-over whatever from the scores on one QT 
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to those on the next OT of the same subject. We hesitate to state this 
in quantitative terms because the time interval between successive exper- 
iments varies widely, and because the change of attitude and general 
psychophysiological condition may remain fairly constant for two or 
three experiments, even when distributed over a considerable period, 
or may change rapidly between experiments done one after the other in 
a single day. 

There are several conventional ways of indicating the statistical 
meaning of the covariance effect for the first thousand QTs. The first 
is the ordinary correlation procedure, which we have already mentioned. 
A second is to Chi-square the data, disregarding theoretical chance ex- 
pectations, comparing, for example, circle-cross, circle-blank scores when 
color is high, and circle-cross, circle-blank scores when color is low, to 
see if the two sets of circle-cross, circle-blank scores derive from the 
same parent population. A negative finding here would indicate the 
operation of extra-chance factors. A third procedure is to throw out 
scores at the middle of the distributions and simply compare high against 
low (the variable scores when the constant score is low and the variable 
scores when the constant score is high). As Table VIII shows, each of 
these three methods gives a value somewhat higher than ordinarily 
regarded as necessary for significance. We should be the last, however, 
to insist on any such conception as that of absolute reliability. In this, 
as in all other experiments in borderline fields, the problem seems to us 
to be one of accumulating and organizing data in such a way as to 
permit the reader to form his own hypotheses. 


TABLE VIII 
TABLE SHOWING P VALUES OBTAINED BY THREE DIFFERENT METHODS OF ANALYSING 
COVARIANCE BETWEEN COLOR SCORE AND SCORE ON CIRCLE-CROSS, CIRCLE-BLANK TOTAL. 
BASED ON 1000 gts. 








Analysis P 
NS keen Sata nak <p Ok Kp aed a4 .006 
ES Sus Sic oe bas cle tome sep -0004 
ie cade dks wis ndins da ebens .003 








1Color scores when circle-cross, circle-blank total score is less than 48 against color score when circle-cross, 
circle-blank total score is more than 52. 
2Significance of difference between color mean when circle-cross, circle-blank total score is above 52 and 
when circle-cross, circle-blank total score is below 48. 
The data were subjected to a complete Chi-square analysis, in order 
to determine the degree to which each of the various circle-cross and 
circle-blank tasks contributed to the mass trend shown in Figures 12 


and 13. The method was as follows: In the case of each of the criteria 
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(the circle-cross and circle-blank decks, separately and in various com- 
binations) the color scores obtained when the criterion was high (top 
sixth of the distribution of scores) and when the criterion was low 
(bottom sixth of the distribution) were compared by the Chi-square 
method, using the simple fourfold contingency table. The obtained 
results are presented in Table IX. 

The table shows that each of the four decks contributes positively to 
the mass covariance effect, but that the first deck of circles and crosses 
contributes almost not at all, while the second circle-cross deck con- 
tributes to a considerable degree (P — .0004). Also, it is seen that 
although neither of the two decks of circles and blanks contributes sig- 
nificantly to the covariance effect when considered as a unit of 25 cards, 
these same decks, considered as a single unit of 50 cards, contribute to 
the covariance effect to a suggestive degree (P — .006). 

Also, when the total circle-cross and circle-blank tasks are considered 
to be two units of 50 cards each, the covariance gives a P value of .0001, 
the most significant value in the table, whereas when the last four tasks 
are considered as a single unit of 100 cards the P value is .01. 

An important phase of the covariance figure as a whole is seen in 
the material indicating the trend of the low part of the curve. To be 
low on the first task gives a suggestion as to low scores on the last two 
tasks. This type of phenomenon, if genuine, necessitates, as Coover® 
points out, a hypothesis about negative extra-sensory perception. One 
must in a sense know where a target is in order to miss it. This was 
intensively studied by Rhine® in a long series of trials. From the 
present data it seems likely that a slight trace of this negative tendency 
may be one of the results to be expected in routine clairvoyance tests. 

. With due regard to the probable errors, it appears that the covariance 
is present in the work of all major subjects, that is, those who have 
done 60 or more QTs, but it is not clearly present, as far as can be 
seen, in the minor subjects. At the present writing it is uncertain 
whether the minor subjects will in time show the covariance in accord- 
ance with the trends shown in the curves of Figures 12 and 13. It is 
also possible, of course, that the subjects who drop out are among those 
who, for some reason or another, lack the particular psychological dis- 
positions upon which the covariance ultimately depends. It would be 
premature to define the cause of this difference between major and minor 
subjects. When once the covariance appears, however, it seems to con- 
tinue rather steadily, as will be seen later. 


*Coover, J. E., This issue of the JourNAL, p. 17 ff. 
* Rhine, J. B., Extra-Sensory Perception, 1934, p. 62 ff. 
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TABLE IX 


RESULTS OBTAINED BY CHI-SQUARE ANALYSIS, COMPARING COLOR SCORES OBTAINED 

WITH UPPER SIXTH OF DISTRIBUTION OF CIRCLE-CROSS, CIRCLE-BLANK TOTAL SCORES 

AGAINST COLOR SCORES OBTAINED WITH LOWER SIXTH OF DISTRIBUTION OF CIRCLE- 
CROSS, CIRCLE-BLANK TOTAL SCORES. 








Task compared with color! Mean 1? Mean 23 x? P 
1 First circle-cross (a).......... 12.29 12.32 01 .92 
2 Second circle-cross (b).....:.. 11.91 12.79 12.82 .0004 
3 First circle-blank (c).......... 12.22 12.54 ae -32 
4 Second circle-blank (d)........ 12.22 12.51 .14 .29 
f° | See 12.16 12.53 9.50 -002 
is schaekn nse vechat 12.12 12.61 12.30 .0005 
ns ek anadoeeekned 12.09 12.54 6.75 .009 
SES. one cuics sae sees 12.22 12.53 3.03 .08 
9 Total circle-cross (e).......... 12.05 12.79 6.65 .009 
10 Total circle-blank (f)......... 12.15 12.92 7.36 -006 
2 oe RRA ee 12.10 12.85 14.03 .0001 
12 Total circle-cross, circle-blank. . 12.14 12.81 6.39 01 

















1In rows 1 to 8 the tasks are considered as separate units of 25 cards each. In rows 9 to II the circle- 
cross and circle-blank tasks are considered as separate units of 50 cards each. In row 12 the circle-cross and 
circle-blank tasks are considered as a single task of 100 cards. 
mean obtained by averaging all color scores obtained with lower sixth of circle-cross, circle-blank 
total distribution. 
r mean obtained by averaging all color scores obtained with upper sixth of circle-cross, circle-blank 


total distribution. 

Some of the data since the first thousand, which cannot yet be ad- 
equately reported, suggest that too much knowledge of one’s own results 
may destroy the covariance effect, at least in certain subjects; this is a 
separate question with respect to which we are not as yet ready to 
report. 

When there is reason to suspect that a subject is manifesting, during 
successive QTs, a more or less consistent tendency to covariance be- 
tween color scores and circle-cross, circle-blank scores, we need a unit 
by means of which to measure the trend and ascertain its significance. 
Such a unit is provided by the product, in sigma units, of the deviation 
from chance on one task and the deviation from chance on another task, 
with due regard to algebraic sign. The theoretical sigma of the color 
score, where the number of calls is 25 and the chance expectation is 
12.5, is 2.5 ( Vnpq), and since the sigma of the circle-cross, circle-blank 
score, where the number of calls is 100 and the chance expectation is 
50, is 5, a subject who made a score of 15 on color, and 55 on circle- 
cross, circle-blank would be one sigma above chance on each, and the 
resulting product would be 1 x 1, or one positive unit. 

Similarly a subject with a score of 15 on color and 60 on circle-cross, 
circle-blank would be 2 units above chance; a subject making 15 on 
color and 45 on circle-cross, circle-blank would be one unit below chance, 








Tas 
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in terms of this product of a positive and a negative quantity. When 
color and circle-cross circle-blank scores are both less or both greater 
than chance expectation we have a positive value in the covariance 
score; when one is above chance, the other below, we have a negative 
covariance score. This procedure is simply a phase of the ordinary 
computation of the product-moment correlation. But it has the value of 
revealing the general trend of a subject over a large number of QTs. 
All that need be done is to ascertain the positive covariance, and the 
negative covariance, subtract the latter from the former, and ascertain 
the significance of this residual positive covariance. 

In Figure 15 are presented cumulative covariance curves for the 
five major subjects, plotted as just described. Each point on the abscissa 
is the number of OTs completed at that point; the algebraic sum of all 
covariance shown at a given time is plotted on the ordinate. On this 
basis a steadily mounting curve means consistency in the manifestation 
of the covariance effect ; an irregular curve, cutting the base line fre- 
quently or remaining near it would mean the absence of a clear co- 
variance effect. The curves shown in Figure 15 clearly indicate the 
degree to which the covariance appears for a given major subject in all 
work up to any given point (since the points on the curve are averages 
of 5 values, terminal values, if used to compute r, should be multiplied 
by 5.) The significance of each of these curves can be determined from 
the values presented in Table VI. The values in the P column represent 
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Ficure 15. CUMULATIVE COVARIANCE CURVES FOR FIVE MAJOR SUBJECTS. 
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Ficure 16. CUMULATIVE COVARIANCE CURVES FOR TWO MAJOR SUBJECTS. 


the probabilities that the individual rs for each subject are due to chance; 
since each r is determined by the calculations involved in plotting the 
curves in Figure 15, the P values in Table VI give the probabilities that 
the final residuals of positive covariance, represented by the last points 
on each of the curves, are chance effects. The control curve was ob- 
tained using scores of targets checked against succeeding targets. 


THE PROBLEM OF LINKAGE 


It has been pointed out'® that in order to determine whether the 
covariance effect is genuine it is necessary to investigate the possibili- 
ties of linkage in the targets called by the subjects. Any linkage exist- 
ing between two parts of the same target, any tendency, for example, 
for a red playing-card in the playing-card deck to be accompanied to an 
extra-chance degree by a circle in the circle-blank deck would con- 
tribute toward covariance. That is, if the subject consciously or un- 
consciously responds to such a linkage in the targets, and tends to adopt 
that linkage in his own calls, obtained scores will show covariance. If 
the linkage in the target is such that red playing-cards in the one deck 
tend to be accompanied by circles in the other deck, and if the sub- 
ject’s calls show the same linkage, both scores, i.e., the subject’s scores 
on the two decks will be either abnormally low or high. Therefore it 
is necessary to investigate the targets to see if any such linkage actually 
was present, in spite of the fact that in much of the work the subjects 
never did have access to records of the targets and hence could not 
normally have obtained information as to the presence of any such link- 


1° See Letters and Notes, this issue, page 118. 
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age. The 140 targets called by subject RF (the most striking case of 
covariance) were subjected to such a test. The method was as follows: 

1. Linkage between circle-cross and circle-blank decks. Each of the 
140 targets called by RF contains a deck of 50 circle-cross cards and a 
deck of 50 circle-blank cards. It was desired to determine the linkage, 
or association, between the two decks, to determine, in other words, 


- whether the presence of a circle (or a cross) at any point in the circle- 


cross deck had any predictive value as to the presence of a circle (or a 
blank) in the same position in the circle-blank deck. The presence of 
such association may readily be tested through the use of the four-fold 
contingency table: 


Circle-cross deck 
Circle Cross 





Circle 


Circle-blank 
deck 





Blank 














We have two decks of cards, then, each of which has attributes A 
(circle) and B (cross or blank). If there is no relation between the two 
attributes, we expect to find the same proportion of A’s among the 
B’s as among the non-B’s. The fourfold contingency table, as shown 
above, provides the required information; through the use of such a 
table, Chi-square is obtained, which may be translated into terms of 
probability (P) that such a result is to be expected through the opera- 
tion of chance alone. 


The 140 targets called by RF were broken down with respect to 
different shufflers (experimenters), and the results obtained are pre- 
sented in Table X. It is seen that the P values obtained are well within 
the limits of chance, so that no known linkage obtains between these 
two parts of the target. 


As will be seen later, it is necessary to consider the direction of any 
linkage which might be present. Thus we may arbitrarily define as 
positive the tendency for a circle in the circle-cross deck to be accom- 
panied by a circle in the circle-blank deck, as negative the tendency for 
a circle in the circle-cross deck to be accompanied by a blank in the 
circle-blank deck. 
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TABLE X 


P VALUES OBTAINED IN MEASURING AMOUNT OF ASSOCIATION BETWEEN CIRCLE-CROSS 
AND CIRCLE-BLANK DECKS IN 140 TARGETS CALLED BY RF. 








Shuffler N P Nature of linkage 
hs cetelennwe 58 18 Positive 
SA 19 .48 Negative 
ids o Uidcky 63 58 Negative 

Pe thnnzes on ot 140 .82 Positive 














*On eight occasions ET’s schedule made it physically impossible to be on hand at a stated time (10 AM or 
5 PM) for the shuffling of a ay Moy oe target. On these occasions other students, trained in the procedure, 
did the shuffling. We regret that these exceptions occurred in the routine. The eight targets differ in no 
evident way from the remaining 55 targets prepared by ET, nor were the hit or covariance scores out of the 
ordinary range. Since we are making it a rule to exclude no records, these are pooled with the ET series. 


It is of interest to note that the linkage which is present is reliable 
for none of the series, and also is inconsistent, the P value for the en- 
tire 140 targets indicating almost no residual association, the various 
positive and negative trends cancelling each other when combined. 

2. Linkage between playing-cards and circle-cross and circle-blank 
decks. The test for this linkage was the same as previously described ; in 
this case the degree of association between the playing-card deck and 
each of the circle-cross and circle-blank decks was determined. The 
same targets were used—the 140 called by RF. The obtained values 
are presented in Table XI. 


TABLE XI 


P VALUES OBTAINED IN MEASURING AMOUNT OF ASSOCIATION BETWEEN PLAYING 
CARDS AND CIRCLE-CROSS, CIRCLE-BLANK DECKS IN 140 TARGETS CALLED BY RF. 








Shuffler N O0+1 O+2 Oo—1 Oo—2 
IRS 58 -17 (+)* 59 (—) .60 (+) .40 (+) 
ae 19 -29 (+) .03 (—) 1.00 (0) .84 (—) 
on ea 63 .43 (—) .25 (—) -67 (+) .79 (+) 

ee 140 .45 (+) .06 (—) .53 (+) .52 (+) 




















*The algebraic signs in parentheses indicate the nature of the association. (+) indicates a tendency for a 
red playing-card to be accompanied by a circle, (—) by a blank. 


Here again the P’s are not of great significance, although in the case 
of the second circle-cross deck the nature of the association is consist- 
ently negative. More will be said of this later. 

It was thought desirable to determine the same linkages for the 
calls of RF. Using the same methods, then, these computations were 
made, the results being presented in Tables XII and XIII. 

Comparing Tables XI and XIII it is seen that although the linkages 
for the calls are generally more pronounced than the linkages for the 
cards, the total values do not approach the point of complete reliability. 


| 
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TABLE XII 


P VALUES OBTAINED IN MEASURING AMOUNT OF ASSOCIATION BETWEEN CIRCLE-CROSS 
AND CIRCLE-BLANK calls oF RF. 




















Shuffler N P Nature of linkage 
oie 58 .04 Positive 
3a 19 .26 Negative 
a 63 .04 Positive 

airs 140 .02 Positive 
TABLE XIII 


P VALUES OBTAINED IN MEASURING AMOUNT OF ASSOCIATION BETWEEN PLAYING-CARD 
AND CIRCLE-CROSS, CIRCLE-BLANK calls oF RF. 








Shuffler N O+1 O+2 Oo—1 O—2 
pS ie 58 .27 (+) 73 (—) .19 (—) 44 (+) 
RP 19 .07 (+) .58 (+) .14 (—) .78 (+) 
ee 63 .89 (+) -02 (+) -11 (+) -26 (+) 

ee 140 -13 (+) -17 (+) .29 (—) 17 (+) 




















An interesting fact which emerges from the tables is that while the 
greatest association between playing-cards and the other decks is be- 
tween playing-cards and the second circle-cross deck (Table XI), and 
is negative, the linkage in the subject’s calls on the same two decks is 
positive. Such a finding tends to make the covariance effect the more 
striking, since it is obtained in the face of a linkage in the direction 
which would produce negative covariance, whereas the actual covariance 
of the subject’s scores was positive. And in general, consultation of 
the various tables indicates that the covariance effect, as demonstrated 
in the present experiment, is probably not attributable to linkages of the 
sort studied, either in the targets or in the subject’s calls. 


DISCUSSION OF RESULTS 


Since we are not dealing with an extra-sensory function spread 
throughout the single QT as a whole, it is incumbent upon us to venture 
a guess as to why task 1 is related to tasks 3 and 4, and not to task 2 
(ESP cards). One possible reason suggests itself; the odds in calling 
ESP cards correctly are 1 in 5, while in calling for color in the playing- 
cards, and in calling the circle-cross and circle-blank tasks, the odds 
against correct calls are 1 in 2. Anyone who calls the QT a number of 
times will see what is meant by saying that the attitude depends a great 
deal upon the odds. If one mixes 5 circles with 45 crosses, for example, 
and asks the subject to locate the circles, using only five guesses, the 
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situation is markedly changed. It is possible that the attitude in calling 
the circle-cross and circle-blank tasks, and in calling the colors, in the 
playing-card tasks, is a consistent “fifty-fifty” attitude; most subjects, 
when they see their scores on the playing-cards, are likely to remark, 
“I got so and so many colors right,” compared always against the 12.5 
chance expectation. This means that the red versus black, or fifty-fifty 
situation is real to them, and the attitude, in terms of fifty-fifty odds, 
can well carry through tasks 1, 3, and 4. 

Now the scores on suit, where the odds are 1 in 4, are consistently, 
for all the data, just about one-half of the scores on color. The co- 
variance of suits against circle-cross, circle-blank is the same as for color, 
the only difference being that the regression lines for suit are slightly 
less reliable than for color, because of having only half as much material 
for comparison (theoretical expection being for half as many suit cor- 
respondences as for color correspondences ). 

The fact that the suit score is half the color score, however, appears 
to mean that suit, as a designation independent of color, is not con- 
tributing to the covariance at all. A comparison of suit scores with 
circle-cross, circle-blank scores gives the same result as a comparison of 
“color minus suit” scores, that is, the scores on color hits which are not 
suit hits. This means that suit scores correlate with circle-cross, circle- 
blank scores only because in order to call the suit correctly one must 
call color correctly. Now the suit odds, being 1 in 4, are comparable to 
the ESP card odds of 1 in 5, and as such have little in common with 
the attitude involved when odds are even. When the “set” is for the 
fifty-fifty odds, nothing beyond chance occurs in tasks where the odds 
are other than fifty-fifty. 

In harmony with this interpretation is the fact that “number” scores 
on the playing-card task, that is, correspondence in terms of ace, 2, 3, 
etc., show no covariance with the circle-cross and circle-blank tasks. The 
odds here are 1 in 13. Also, direct hits on the playing-card task, in 
which odds are 1 in 52, show no covariance with circle-cross and circle- 
blank scores. But of course this negative evidence is of slight weight. 

It may well be asked whether a slight ESP function, if present, is 
spread evenly through the single QT or whether it comes by fits and 
starts, as if a faucet or light were turned on and off. This question can 
be answered with fair certainty by analyzing long runs of right or 
wrong guesses. If there is any tendency for a positive ESP function to 
intervene prominently at a given point and to remain present for a few 
seconds, it should produce a string of correct calls; similarly a negative 
factor of prominence should cause a string of incorrect calls. 
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The 25 playing-cards, the 25 cards in each box of circle-cross cards 
and in each box of circle-blank cards were broken into arbitrary units 
of 5 cards, and the total number of units of 5 correct and 5 incorrect 
was tabulated for the thousand QTs. The chances of getting a unit of 
5 all correct, where odds are 1 in 2 per call, is of course approximately 
1 in 32, so that for the entire thousand QTs the number of units of 5 
right should be 125,000 divided by 32. The empirical data agree closely 
with this figure. So also with the boxes of all-wrong guesses. There is, 
so far as we can see, no bunching of hits, as tested by this criterion. It 
seems to us that whatever is present which is favorable or unfavorable 
carries rather evenly through a single QT. 


A TENTATIVE PsyCHOPHYSICAL HypoTHESIS 


On general psychophysical grounds the way in which such a func- 
tion, whenever present, would be conceived to work would be as follows : 
The tendency to make a given call, let us say a circle when one con- 
fronts the circle-cross task, is variable, depending upon many factors, 
in a word, the whole set or disposition of the moment. Sometimes the 
tendency to call a circle is very strong, the tendency to call a cross much 
weaker. In other instances the tendencies to circle and cross may be 
almost evenly balanced. A very faint ESP function would be of little 
or no consequence in the former instance, but of considerable con- 
sequence in the latter instance. Thus when the bias for circle is strong, 
a weak ESP function indicating cross could accomplish nothing in this 
call, even if present in maximal degree, and of course an ESP function 
indicating circle would also make no objective difference, when the 
card actually was a circle, i.e., if an ESP tendency is in harmony with 
the prevailing disposition it can accomplish nothing objectively, and if 
it is competing against a much more powerful force it can also accom- 
plish nothing. It is therefore only when the tendency to circle and cross 
are about evenly balanced that we can get any expression of an ESP 
function. If, however, the factors affecting normal thresholds are 
normally distributed, we should expect the raw score in a given task 
to depend first upon the accidental correspondence of associative habits 
and target orders, second upon the amount of ESP present through the 
QT in question, and third upon the exact way in which the ESP func- 
tion is able to overcome contrary associative habits so as to “trip the 
triggers” of the various calls in which the subject is almost evenly 
poised between two tendencies. 


Since the ESP function could on this basis manifest itself only 
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very infrequently in the course of a series of calls, it would not be 
likely to reveal itself in both the even-numbered and the odd-numbered 
elements of a short series, so as to give rise to an appreciable odd-even 
reliability. We have attempted several mathematical analyses to predict 
the odd-even reliability of tasks of 25 and 50 calls when ESP contributes 
only slightly to the variance. The resulting values are so low that it is 
doubtful whether they can be interpreted; they tend to be only slightly 
greater than the theoretical PIs of very low correlations with this 
material, and therefore incapable of giving clear statistical meaning. We 
doubt the value at present of a detailed mathematical analysis of the 
conventional odd-even reliability of scores when the covariance effect 
is of the present low magnitude. But we shall be glad for suggestions 
as to meaningful ways of going beyond the sheer covariance effect as 
such. 


It is perhaps of some importance here to consider the problem of 
maintaining the subject’s morale throughout the long series of trials 
required. The present writers have found it easiest to maintain morale 
in the case of group experimentation, in which a small and congenial 
group of subjects calls the QT in a more or less competitive manner. Of 
importance is the immediate checking of call sheets, which enables the 
subjects to know how they are scoring, and gives them a definite sense 
of contact with the experiment, in which case interest is generally main- 
tained at such a level that subjects wish to go on with the experiment. 
If a subject scores high, he generally wants to see if he can do it again, 
and if he scores low he immediately wants a chance to do better. 

The subjects must not be informed of the covariance effect ; the ex- 
perimenter may, by manifesting interest in other aspects of the calling, 
focus the attention of the subjects upon some other phase of the data 
than the covariance. 

In the present research, including some material beyond the first 
thousand QTs, it appears that in some of the work done under condi- 
tions other than just described the covariance is not present to the degree 
indicated in the group experimentation. It is possible that the type of 
morale generated by group work is considerably more favorable for the 
manifestation of the covariance effect than are conditions in which sub- 
jects are isolated, but we are not yet ready to state this in quantitative 
terms. We do wish to emphasize, however, that the covariance effect is 
apparently, to some extent, a function of the morale of the subject. A 
specific testing of this hypothesis is at present in progress. 
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CONCLUSIONS 


Two defensible interpretations of our data appear conceivable, either 
(1) the covariance effect may be a genuine result with certain subjects 
working under certain conditions, or (2) it may be an extremely unusual 
and at present obscure statistical artifact. Which one, if either, is the 
correct interpretation can only be settled by further investigation. 

At present our chief interests are (1) in discovering whether we can 
repeat the covariance results with new subjects and new methods, (2) 
in testing covariance results with the machine described on p. 11, in 
this issue, and (3) in getting other laboratories to repeat the experiment, 
either by calling their own targets or by calling machine-set material at 
Columbia, keeping their own records and, if convenient, sending us 
daily duplicates. As a final word of warning, those repeating the exper- 
iment are urged, for reasons already mentioned, to keep subjects in 
ignorance of the covariance effect as reported in the present paper. 
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FURTHER DATA FROM A CASE OF HIGH 
SCORES IN CARD GUESSING 


BERNARD F. RIESS 
Hunter College 


N UMEROUS REQUESTS for the complete records and for a more ade- 
quate description of techniques have been received by the author since 
the publication of the original data (3). Kennedy (1, p. 95) has sug- 
gested that in view of the extremely high scores “full determination of 
the conditions under which the unusual results occur would seem to be 
of paramount importance.” In the same citation, it is stated that “no 
information as to when and how the check-up for number of hits was 
made is included in the experimental report.” It is the author’s hope that 
some of these queries and criticisms will be answered in the present 


paper. 

It will be recalled that the subject, in a series of 1,850 trials made 
1,349 correct guesses at the symbols on the usual pack of ESP cards. 
The conditions under which the experiment took place were as follows: 
The experimenter seated at his desk exposed, at 9:00 p.m., the top card 
from the deck which had just been manually shuffled and cut. This card 
was placed face up to one side of the deck which was kept face down. 
At 9:01 the second card was similarly exposed. Immediately after ex- 
posure of each card, its symbol was written in the card column of a 
previously prepared mimeographed record sheet. In this fashion 50 
trials or two shufflings of the pack were run during each of the experi- 
mental sessions. Meanwhile the subject at her home, separated from the 
experimenter by approximately one-quarter mile, wrote her guesses in 
the call columns of her record sheets. The first guess was made at 9:01, 
the second one minute later. During the day following the session, the 
subject’s records were mailed or delivered to the experimenter. The 
card records were kept in a desk drawer. As a rule the sheets were com- 
pared during the week-end. Checking was done by noting correct and 
incorrect responses on the experimenter’s record sheets. Rechecking was 
performed on the subject’s sheets. No witness was present during the 
checking, nor were the card records locked up during the day. This 
constitutes a serious criticism to which the author can only respond by 
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stating that to the best of his knowledge no one at his home interfered 
with the records in any way. Furthermore, the sheets show no erasures 
and are all in the same handwriting. Photostatic copies of the originals 
are available for inspection if desired. The complete call and card rec- 
ords for the 1850 guesses are given in the appendix, arranged in two 
columns for each of the 74 trials. The upper line in each trial contains 
the records made by the subject and the lower line the experimenter’s 
records, calls and cards, respectively. Correct guesses have been indicated 
by upper case letters in the call line. The division into units of five has 
been made for legibility. There was no experimental hiatus between suc- 
cessive fives. The symbols on the cards are represented in the table by 
the following letters: star, s; cross, p; waves, w; square, r; circle, c. 

Lund (2) has recently published data on the attention value of the 
various symbols. He found that the star had twice the attention value 
of either the cross or the rectangle. Other writers have mentioned the 
possibility of symbol predilection. When the data of the 1,850 trials are 
studied, it is seen that the star was used by the subject 366 times, the 
square 367 times, the circle and the waves each 372 times and the cross 
373 times. These differences are obviously not significant. 

Among the questions frequently asked in ESP work is whether the 
subjects use any system of guessing. It would seem apparent that a 
sophisticated subject would try to use each of the five symbols five 
times in each trial. This has, however, been found not to be the case 
in long continued trials. Of the 74 trials in the experiment, only 19 
contained calls in which each symbol was mentioned 5 times. Ten of 
these 19 occurred at the beginning of the experiment. It seems possible 
to conjecture that the subject gave up systematic guessing after the 
initial period of the experiment. Table I gives the distribution of the 
frequency of occurrence of each symbol within a trial. The table is to 
be read as follows: There was one trial in which only one guess was 
given for a square (R). There was one trial in which only two squares 
were given as responses. 


TABLE I 
Frequency of Occurrence of Symbols within a Trial 
No. of symbols Frequency 
in a trial R P W S 
1 1 0 0 0 0 
2 1 2 0 0 0 
3 2 3 5 2 6 
4 12 12 11 15 16 
5 41 40 41 41 33 
6 15 10 11 11 16 
7 1 5 6 5 2 
8 1 1 0 0 0 
9 0 1 0 0 1 
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In accordance with common test procedure, the consistency of the 
subject’s scorés was determined. An odd-even reliability coefficient of 
.74 uncorrected was obtained for the subject’s calls. When the Spearman- 
Brown formula was applied, this rose to .85. This indicates high 
internal consistency of response and seems to show that the factors 
making for high success in card guessing were present throughout the 
entire 74 trials. 


In conclusion, study of the data reveals that, over the whole series, 
the symbols had approximately the same attention value or, expressed 
in other words, the subject had no definite symbol predilection. The re- 
liability of performance in this task was high as indicated by an odd- 
even reliability coefficient of .85. 
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werew pwcecew srrer pcwep srpps 
2. pwWw ss rpepe sSwwe *ePrres RecePC 
swwee wrerwrer wepres pPpeee rpepe 
3. ec sRWC sRWCr PRW rp pweSp pewees 
scrwe wrwep Prwes wppec “Es oe 
4. *RpRe P pRCp wesReC wsSwR wcC pw 
es £¢3 9D pseres spwre pwewr cscww 
4 Cewps CWRRe RSCPS sRWww rPeere?Pr? 
ccepss ecwrere rscps wrwes PWwpwp 
6. PRrerpp S<e28 8 pwwCR cswer CwwpR 
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> eté et PWwC p epRSr SWRCR p SWRw 
rece s pPpwpew psrep swreer wewrp 
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ccswr pwrwp ccepss srwpp rewer 
9. CSRCP WRSCW PwSWre CrwRe wePSP 
e€eresp wresew prewes cwprr cwp sp 
10. CCWRW SPPCR wWS rR PSCSR PS pWe 
‘ ccwrw spper rwewr pecesr pscwp 
11. PrrwW eCwCc esr P prt eR epRess RpwsS'r 
ppwrw rewee psepwr serep rewee 
12. c rWP 8 WC sPR pwSPS PRweS RR pwwWw 
wpwpw weepr crepe presses cesce 
13. SP swW P SWW ce RC pw S- pCRRC SPRSC 
sprpw pswww rewps ecrre spree 
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14. RRPCW WspPcc WwWRP SS SRW CPRSP : 
rrpew wspec wewrp sssekfrw cprep 
15. SWRW er RW pSR eCCcrR Pr? sa PP SSCRS§ ‘ 
sewrww rwwer peccr Ppcpp escrs 
16. PPRSW SRRRC wCSPC c SWPR WwCWe 
pPpprsw 6 @¢ £ £4 scape psewpr wecww 
17. PRWWW SscecsS WRWRP SPCRC RPrPScece 
prwww scspes wrwrep sperce rpsecec 
18. WPPWR sccss RSP cC RPC SW RPWCW 
wppwrer scceses rspre rpesw rpwew 
19, CWS SR PCCRC WPWPS CPRRS PSRWW 
cwsesr pccere wpwps cpreres Psetrww } 
20. PP «FPR SWCWR SCPSS CWCSR RWRPC 
Pppwpr swewr scpss cwcecer rwepe | 
21. CPR SW WRWRP CSWwR sC SCW RP ity 
cprsw wrewerp cswer pcscw rppes 
22. sC SPC WSCCR WPPps RsS  rW CSWRp | 
wespe wseer wppres rpesepw cswre | 
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weresee wwesp prwer sspwr pserep 
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cwwre csrps swwep rppes wspee 
29. WsSsSsS R pWcC RsrrR epPCs RP pwW 
wesss rewese rwppe rppes rpwew } 
30. WpwwP PWccep swpCP cS pPc RP SwW 
wececcp pwrer wescp rswpe rpssw 
31. cP pRS WPwPW RRWPC SRCSc pCS rc 
rFpers wpespw rrwpe srcsw wespe 
32. WWCPw SSRRP PCPWS SRRcC S pCWp 
wwepp ssrrp pecpws srrre swewe ) 
33. PWCWC PCWCR RSPWR s8s8& P RSR cW | 
pwewe pewer rspwe capsp rsersw | 
34. SRSRP PSSwW RCWRC SWPCC PWWRP 
srsrp psscew rewre swpec pwwerp 
35. ec sW SC rCCRW SRR PW CCwS&ec PR SWP 
rwweec sccrw srrpw c cp sp prswop 
36. P rSRC ec SPWR CwPWs WpsS pC c r RW P 
ppere rspwe cs pws weesere werwp 
37. CPRWS R rRWS PRPSC RPCCW sSwWs | 
cprws rprwes prpse rpcecw wscew s 
38. SSPWW CpPRS RSCwR pW PwWw PPRCW 
sspww csepres reccr cwpnrw pprcw 
39. WWwCr PW sCP RPé&e?P §S cRW ceceC Pes 
wwesce pwrep rpssp sswerew crcepr 
40. wSCPC RRswe PcWSc CRPPR WWpess } 
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SYMPOSIUM ON ESP METHODS, SOUTHERN 
SOCIETY FOR PHILOSOPHY AND 
PSYCHOLOGY* 


INTRODUCTION 


J. B. Rune, Chairman 


= AND GENTLEMEN: This is the third symposium on extra- 
sensory perception held in connection with psychological meetings in 
this country. The first was held in Madison, Wisconsin, last spring; 
and the second at Columbus, Ohio, last fall. This is the first of which 
the chairmanship and arrangement of the program were left to one 
who has reported evidence in favor of the hypothesis of extra-sensory 
perception. Whether this is evidence of evolution or merely Southern 
courtesy, I appreciate the invitation of the Society to take this part in 
today’s program. 

There has been of necessity a short time for the preparation of this 
symposium program. The papers have, I believe, all been prepared 
within the last week or ten days. There was not time to secure repre- 
sentation from certain points of view that might further have enriched 
the variety of the program. The arrangement that has been made is as 
follows : | 

We will, if there is no serious objection, incorporate Dr. Pratt’s 
paper into the symposium program. The first part of the discussion will 
be on the mathematics of probability dealing with the problems of ESP. 
Two short papers will be given under this heading: the first by Dr. 
T. N. E. Greville of the Department of Mathematics, University of 
Michigan, entitled “Past Developments of the Mathematics of Prob- 
ability with Special Relation to Problems of ESP.” This will be fol- 
lowed by a paper by Dr. Joseph A. Greenwood of the Department of 
Mathematics, Duke University, on the topic “Some Unsolved Problems 
in the Mathematics of Probability Arising from the ESP Research.” 


*This symposium was a part of the program of the Thirty-fourth Annual 
Meeting of the Southern Society for Philosophy and Psychology, which met at 
Duke University and the University of North Carolina, April 7 and 8, 1939. The 
original papers are published in full here, with permission of the authors, and a 
stenotyped record of the discussion is included. 
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Next will come the experimental section, initiated by a paper en- 
titled “A Further Advance in Methods of Investigating Extra-Sensory 
Perception,” in which Dr. J. G. Pratt, of the Parapsychology Labora- 
tory of Duke University, will review briefly the development of methods 
and give the results of his own experiments. Finally will come a paper 
by Mr. Charles E. Stuart of the Parapsychology Laboratory of Duke 
University, entitled “Some Unsolved Problems of Methodology.” 
These papers will be followed by a general discussion. 


In addition to these papers, it was desirable, I thought, to have a 
representation of the various points of view among the critics of the 
research in extra-sensory perception, and accordingly I wrote six of 
the most active of these critics located in the Southern half of the 
country, inviting their presence and co-operation in discussion, and 
asking them, in case of their inability to attend, to submit a brief paper 
to be read presenting their principal criticisms for consideration here, 
I have just now received a manuscript from one of these, Dr. Lemmon, 
of Washington University, and we shall hear this paper before open- 
ing the general discussion. 

There is no attempt made in the arrangement of the program to 
marshal evidence on the subject of ESP. With the exception of Dr. 
Pratt’s paper—which was a part of the regular program of the Society 
apart from the symposium—the question of evidence is not gone into. 


A SUMMARY OF MATHEMATICAL ADVANCES 
BEARING ON ESP RESEARCH 


T. N. E. GRreEvILLE 


The current investigations of extra-sensory perception have sug- 
gested certain mathematical problems of great interest and a few of 
considerable difficulty which have been occupying the attention of a 
number of mathematicians in recent years. It is the purpose of this 
paper to summarize the progress which has been made in the solution 
of these problems. I shall discuss the subject under the following heads: 
first, a brief reference to the history of the subject; secondly, an enu- 
meration of the results which have been obtained in connection with 
the more fundamental questions of the application of statistical and 
probability theory to data obtained in ESP investigations; and lastly, 
some mention of a few special problems which have arisen in particular 
phases of the work. 
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HIstTory 


One of the alternative hypotheses which must be met in connec- 
tion with any set of experiments purporting to demonstrate extra- 
sensory perception is obviously that of chance. Of all such possible 
hypotheses it is, in a sense, the most difficult to refute as it cannot be 
ruled out by any experimental precautions. It is possible only to com- 
pute the probability of a deviation from the normal situation as large 
as or larger than that actually obtained in a given case; and the deci- 
sion as to whether the chance factor has been effectually excluded must 
be reached, in the final analysis, by a subjective process. 

Nevertheless, the probabilities arrived at in connection with many 
of the reported ESP experiments have been of such magnitude as to 
raise an overwhelming presumption against the chance hypothesis in 
the mind of any reasonable person. It was natural, therefore, that the 
methods used in obtaining these figures should be subjected to close 
scrutiny. The book Extra-sensory Perception was closely followed by 
a series of criticisms of the mathematics used by Rhine and his collab- 
orators, appearing, for the most part, in various psychological journals. 
Most of these unfortunately were ill-considered and wholly or partially 
without justification. Certain of them did, however, point out one fea- 
ture in the calculations employed which was subject to criticism from 
a strict mathematical standpoint, although negligible in its practical ef- 
fect. In all subsequent work the proper correction for this factor has 
been applied. 

When certain of these criticisms began to be heralded in popular 
articles and even in some scientific circles as disproofs of the claims of 
extra-sensory perception, a number of mathematicians became aroused. 
They properly felt that, whatever might be the truth about ESP itself, 
the public should not be misled into believing that the experiments with 
ESP cards could be refuted by such unfounded assertions about the 
mathematical procedures used. As a result, much attention was given to 
the subject at the Indianapolis meeting of the Institute of Mathematical 
Statistics in December, 1937, culminating in the now well-known state- 
ment by Professor Burton H. Camp (2), then president of the In- 
stitute. This statement seems to have been generally accepted and at 
present the basic mathematics involved in this work is being challenged 
only by a small group of psychologists some of whom, so far as one may 
judge by their writings, are without extensive mathematical training. 
It is fair to say then that the period of heated controversy over the 
mathematical aspects of ESP research is at an end. Obviously, this does 
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not in any sense imply that the existence of extra-sensory perception 
has been proved, but merely that if fault is to be found with the re- 
ported work it must be, as Camp has said, “on other than mathematical 
grounds.” 

This controversial period had a great value in that it called atten- 
tion to many interesting mathematical problems which might other- 
wise have escaped the notice of mathematicians at large. Much credit 
is due to Greenwood and Stuart for so clearly laying the groundwork 
for these studies in two articles in the JoURNAL oF PARAPSYCHOLOGY 
(8, 9). They suggested certain problems, which in turn suggested oth- 
ers, and now the problems which have arisen make up an imposing 
list. Some of these have been solved and some, as Dr. Greenwood will 
presently tell you, have not. These developments have formed the sub- 
ject of at least ten technicai mathematical papers, not to mention several 
now in preparation and perhaps others which have not come to my at- 
tention. 

Basic RESULTS 

Coming now to the enumeration of the results obtained as regards 
basic principles, Greenwood and Stuart have shown (8) that the same 
conclusions would be drawn from all published experiments (except 
perhaps a few borderline cases) no matter which of several recognized 
mathematical techniques is applied to them. They have shown also (8) 
that the average or expected number of correct guesses (or “hits”) 
with the standard ESP deck is 5, and that this is not affected by the 
particular pattern of the subject’s call series. They have also drawn 
the important distinction between the “binomial” hypothesis in which 
complete independence of calls within the subject’s call series is as- 
sumed, and the “matching” hypothesis in which it is assumed the sub- 
ject will call each symbol exactly five times. Inasmuch as neither as- 
sumption is, as a rule, completely realized in actual experiments, Green- 
wood has further shown (4) that the variance of the number of “hits” 
is a function of the subject’s call series; and that it has its minimum 
value of zero when the subject chooses to call the same symbol all the 
time, and reaches its maximum under the conditions of the matching 
hypothesis. That hypothesis, therefore, represents the most conservative 
assumption and the variance based on it may always be used with per- 
fect safety. Generalizing this result, he has given (5) a formula for 
the variance of “hits” with an m x n target deck and any pattern of the 
call series. 

Stevens (16) has shown that if the target deck does not contain 
equal numbers of the different symbols, the mean, as well as the 
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variance, depends on the subject’s call pattern ; and he has given formu- 
las for both mean and variance under these general conditions. 

Huntington (11) has indicated the usefulness of the Charlier se- 
ries in obtaining the probabilities for short runs, where the unadjusted 
probability integral is not sufficiently accurate. 

In a thorough analysis of a gigantic chance series, Greenwood (6) 
has verified empirically the basic characteristics of the chance distribu- 
tions as previously demonstrated by mathematical analysis. This is 
most important as some writers had questioned whether the assump- 
tions made in probability theory are realized with actual decks of cards. 
He found also that the variance associated with the call series of most 
subjects closely approximates that of the binomial. 


SPECIAL PROBLEMS 


Greenwood and Stuart (8) proposed the problem of computing the 
entire frequency distribution for the matching hypothesis. Huntington 
(10) computed the entire distribution for 3 x 3 and 4 x 4 decks and 
Sterne (14) obtained exceedingly close approximations to the 5 x 5 
matching frequencies by calculating the first four moments and fitting 
to them a Pearson Type I curve. Subsequently, Professor Bancroft 
Brown (17) and I (3) independently arrived at the exact frequencies. 
My method is applicable to the general matching problem considered by 
Stevens. 

Olds (13) has developed a most ingenious method for obtaining the 
moments, and theoretically the complete frequency distribution, for the 
matching of two arbitrary decks. However, for the latter purpose, 
his method is not of practical use. 

Bubb (1) has given an accurate formula for combining “critical 
ratios” (i.e., the ratio obtained by dividing the deviation of the ob- 
served number of “hits” from the expected number by the standard 
deviation) which is preferable to the empirical formula suggested by 
Rhine (13, p. 34). 

A more difficult problem, proposed by Greenwood (7), has been 
called the problem of “optional stopping.” It is noted by ESP experi- 
menters that the performance of a subject in routine ESP tests has 
shown a tendency to decline with use. For this reason, some experi- 
menters would like to be permitted not to specify in advance the length 
of a set of experiments, but to terminate it optionally when experimen- 
tal conditions seem to make this step advisable. Under these condi- 
tions, what probability should be associated with a given “critical ratio” ? 
Alternatively, the problem may be stated thus: “What is the probability 
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that at some time in the course of a series of indefinite length, a given 
critical ratio will be attained ?” 


Greenwood and I (17) have shown independently that if there is no 
upper limit to the length of the experiment, the probability in question 
is unity. It then becomes of interest to know the value of the prob- 
ability when the number of calls is limited to a very large but still 
finite number. This value is difficult to approximate and no one, to my 
knowledge, has succeeded in finding an upper bound which is suffi- 
ciently close to be of practical value, although Greenwood has given an 
empirical estimate on the basis of his study of a chance series of 500,- 
000 calls. I have recently obtained a partial solution of the problem 
by the following scheme. The experimenter is required to select in ad- 
vance one of the following three ranges: 


Range I 1 to 100 runs of 25 (25-2,500 calls) 
Range II 20 to 2,000 runs of 25 (500-50,000 calls) 
Range III 400 to 40,000 runs of 25 (10,000 to 1,000,000 calls) 


He must then make his experiment of a length lying within the range 
he has selected. It can then be shown (17) by suitable mathematical 
analysis that, no matter which range has been selected, a critical ratio 
of 5 will occur less than once in 50 times; a critical ratio of 6 will oc- 
cur less than once in 200 times; a critical ratio of 7 less than once in 
1,250 times ; 8 less than once in 10,000 times; 9 less than once in 100,- 
000 times; and 10 less than once in 1,000,000 times. These figures 
would indicate the adoption as a criterion of significance of a critical 
ratio of 514 as roughly equivalent to the 2% which is customary under 
ordinary circumstances. It should be emphasized, however, that the 
probabilities given above are merely upper bounds, not approximations. 
On the other hand, there appears to be some indication that the usual 
figure of 214 may be too low (7, 12). 

Greenwood has attacked this problem (17) from a different angle. 
Assuming the statistical distribution of ESP scores obtainable by chance 
to be approximately normal (an amply valid assumption for series of 
about 100 runs or longer), suppose that an experimenter obtains in suc- 
cession several blocks of data, deciding the length of each block in ad- 
vance of the execution of that particular block, and recalculating the 
critical ratio after each block on the basis of the total data up to that 
point. The experimenter is further required in advance of the entire 
experiment to agree on a maximum to the number of such blocks of 
data to be obtained, and then must not exceed the maximum so chosen. 
Finally, as the work pregresses, the lengths of the successive blocks 
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must be determined so that the second block is not longer than the first, 
and no subsequent block is longer than the second. Under these condi- 
tions (which perhaps sound complicated, but really are not difficult 
to realize in practice), Greenwood has obtained a method of approx- 
imating closely the probability that any of the critical ratios calculated 
after the successive blocks will be as high as that actually obtained for 
the total data, and this probability would then be a suitable measure 
of the significance to be attached to an experiment carried out under 
the conditions stated. 

After the mimeographed copies of my paper had been prepared, 
Dr. Greenwood pointed out to me the rather evident fact that under 
either my method or his it would be quite proper to take as the criterion 
of significance not the critical ratio for the entire data but the highest 
critical ratio obtained at any optional stopping place. While this may 
perhaps seem a bit “radical” at first, it should be observed that if the 
probability is one in a million of securing a certain critical ratio at any 
point in a series and such a value actually occurs, then something has 
happened which would happen by chance only once in a million times, 
and this calls for explanation just as much as in the case of any other 
type of event. 

Some experimenters have observed a tendency for subjects to pro- 
duce runs of scores both above and below the chance level, rather than 
high and low scores distributed in a purely random manner. Greenwood 
has recently devised (17) a measure of this tendency and has obtained 
formulas for the moments of its distribution, so that it will shortly be 
possible to translate an observed value of this statistic into a prob- 
ability of its occurrence by means of tables which he is preparing for 
that purpose. I am indebted to Dr. Greenwood for the description of the 
two results last mentioned, and it is only fair to say that I have not 
had the opportunity to go over the mathematical developments in com- 
plete detail. 

REFERENCES 


1, ooo W. On the combination of critical ratios. J. Parapsychol., 1938, 2, 


2. a. B. H. Statement quoted in “Notes” section of J. Parapsychol., 1937, 1, 


3. Grevitte, T. N. E. Exact probabilities for the matching hypothesis. J. Para- 
psychol., 1938, 2, 55-59. 
4. GREENWwoop, J. A. The variance of ESP call series. J. Parapsychol., 1938, 2, 


5.GRrEENwoop, J. A. The variance of a general matching problem. Ann. Math. 
Stat., 1938, 9, 56-59. 

6.GrEENwoop, J. A. Analysis of a large chance control series of ESP data. 
J. Parapsychol., 1938, 2, 138-146. 














92 The JOURNAL of PARAPSYCHOLOGY 


7.GreENwoop, J. A. An empirical investigation of some sampling problems, 
J. Parapsychol., 1938, 2, 222-230. 


8. GreENwoop, J. A. & Stuart, C. E. Mathematical techniques used in ESP re- 
search. J. Parapsychol., 1937, 1, 206-225. 


9. GREENWooD, J. A. & Stuart, C. E. A review of criticisms of the mathematical 
evaluation of ESP data. J. Parapsychol., 1937, 1, 295-304. 


10. Huntincton, E. V. Exact probabilities in certain card-matching problems, 
Science, 1937, 86, 499-500. 


11. Huntincton, E. V. A rating table for card-matching experiments. J. Para- 
psychol., 1937, 1, 292-294. 


12. Leuba, C. An experiment to test the role of chance in ESP research. J. Para- 
psychol., 1938, 2, 217-221. 


13. OLps, E. G. A moment-generating function which is useful in solving certain 
matching problems. Bull. Amer. Math. Soc., 1938, 44, 407-413. 


14. Rune, J. B. Extra-Sensory Perception. Boston: Bruce Humphries, 1934. Pp, 
xiv + 169. 


15. — T. E. The solution of a problem in probability. Science, 1937, 86, 


16. Stevens, W. L. The distribution of entries in a contingency table with fixed 
marginal totals. Annals of Eugenics, 1938, 8, 238-244. 


17. Not published. 


SOME MATHEMATICAL PROBLEMS FOR FUTURE 
CONSIDERATION SUGGESTED BY ESP RESEARCH 


J. A. GREENWoopD 


You have just heard a discussion by Dr. Greville of some of the 
ESP problems of a mathematical-statistical nature that have arisen 
and been solved, at least partially. In particular it is interesting to ob- 
serve that the matching problem, alone, has given rise to a considerable 
number of published papers. 

It is my main purpose to give you a description of some of the 
problems which, as far as I know, are yet unsolved. The problems of 
this brief survey rather naturally fall into the three divisions: Selec- 
tion, Miscellaneous, and Optional Stopping problems, and will be taken 
up in that order. 

Many of the problems arise from an assumed characteristic of the 
ESP capacity as, for instance, when it is assumed that there is irregu- 
larity in its occurrence over short intervals. It is then desirable to be 
able to select out for separate evaluation any best consecutive runs 
preassigned or unpreassigned in length. I emphasize that I am not sug- 
gesting that grounds be invented for literally throwing away the other 
data. Perhaps I had better make a general statement of the problem 
covering all such cases..What is wanted is the antecedent probability 
that in the series one would be able to select out a predetermined con- 
figuration having a predetermined property at least as unusual in the 
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probability sense as that one actually selected or observed. That is of 
course nothing but a general statement of the application of probability 
to real conditions. 

Of a more remote nature is the following discussion. Due to the 
very nature of most other statistical investigations the problem of con- 
sidering anything other than the immediate results has not arisen se- 
riously. But with the large number of official and unofficial researchers 
in the ESP field, and the relative rapidity with which large quantities 
of data can be accumulated, the problem of the various degrees of se- 
lection of data must be considered. The above problems involved the 
internal selection of data from a single experiment. On the larger scale 
the problem is how to select one or more experiments from a totality 
of similar experiments. As far as a single selection is concerned the 
familiar 1|— q" suffices, and this has an obvious generalization for two 
or more selected ones with results on the safe side. I will illustrate it. 
Say j experiments out of n are observed to have probabilities less than 
or equal to p. Then from the binomial (p + q)®" one finds the a priori 
probability that j or more of the n experiments would have a prob- 
ability less than or equal to p as a result. But to show the inadequacy 
of the latter, two results with probabilities of .001 and .000001 selected 
from say ten experiments are accorded the same significance as a couple 
as if they had been .001 and .002, respectively. 

This problem of larger selection and interpretation is not confined 
to ESP data alone. It enters, whether or not it is ignored, upon the 
second similar investigation that is undertaken. I think it is obvious 
that the probabilities obtained by whatever refined methods and condi- 
tions must finally be subjected to a common-sense interpretation. Thus 
I feel that the common-sense interpretation applied to an ESP experi- 
ment enters in the form of a reasonablé assumption as to the number 
of similar, potentially reportable or publishable experiments undertaken. 
That is, as I see it, there are two probabilities to be considered by the 
experimenter: First, the a priori probability relevant to the occurrence 
of the phenomenon as observed in his one experiment; the other, the 
meaning of the result viewed as just one of many other attempts. 

Naturally, for his own guidance, one will be more concerned with 
his own immediate results, and this may be a sufficient reason for com- 
pletely ignoring the second suggestion. But for the crucial purpose 
of convincing others that he has something, he will on some occasion 
have to revert to the second consideration. I might add that there are 
several individual ESP experiments which will stand this test for long 
time to come. 
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In the second category a rather peculiar problem arises out of the 
ESP hypothesis. If it is assumed that the scoring level of two given 
subjects is about the same, then when the means of the two series are 
quite close, we should like some measure of their closeness in contra- 
distinction to the usual standard deviation of the difference of two 
means. This measure would appear to be a function of the value of at 
least one of the deviation ratios obtained. 

Of a radically different character is the problem of sequences of 
successes or hits within the single “run.” Professor Uspensky gives, 
in his book on probability, a solution to the problem of sequences of 
successes in a fixed number of trials in the case of independent prob- 
abilities from trial to trial. There is here a clear case of a needed gen- 
eralization to the matching cases, or at least a proof that the assump- 
tion of independence is sufficiently accurate in all the desired cases. 

Pertaining to the ESP shuffle hypothesis arises the complex match- 
ing problem of the distribution of total simultaneous hits obtained by 
matching an ESP deck of cards with several other ordered decks. This 
can be considered either from the viewpoint of unknown random order 
or from known order of the target decks. And from the same hypothesis 
we get what may be a trivial problem of, say, finding the probability 
that a randomly shuffled deck will result in three or more circles being 
adjacent. Several types of correlation problems have arisen recently. 
I shall not give them room for statement in this paper. However, an off- 
shoot of one, the solution of which is unknown to me and the lack of 
which prevents a generalization of another is the following: If x and y 
are identical correlated variables, what is the correlation between x" 
and y"? 

The third type, or “Optional Stopping’ problem described by Dr. 
Greville has, it seems to me, more ramifications and certainly more diff- 
culties ahead of it than the problems already considered. The partial 
solutions given by each of us, while loosening considerably the strait- 
jacket placed on the experimental procedure (i.e., effectively naming 
in advance the number of runs to be undertaken in the experiment), 
falls short of ideal conditions. Naturally, complete freedom in ceasing 
the experiment is desirable provided not too much significance is lost in 
the process. 

Even more freedom is suggested by an optional choice of an a 
posteriori stopping point for evaluative purposes. This is seen to be 
somewhat of a selection problem already stated in which one endpoint 
of the selected sequence is fixed at the beginning of the series. These 
procedures, while ignoring the actual remaining values of the series, 
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must take into account the number of such values. As was indicated 
by Dr. Greville, approximations to these are already solved. 

It is hoped that this paper has given you an idea of the type of 
problems of a mathematical-statistical nature that we encounter in this 
field of research. Perhaps some of you will find these as stimulating as 
several of us have in the past. 


A FURTHER ADVANCE IN METHODS OF TESTING 
EXTRA-SENSORY PERCEPTION 


J. G. Pratr 


This paper will describe a new procedure for the testing of clair- 
voyance. As an introduction it is necessary to survey briefly the develop- 
ment of experimental methods in this field to the present time. 

When we use the term “extra-sensory perception,” we naturally 
think of the investigations which had their beginning at Duke Univer- 
sity about 10 years ago. However, organized work in the field gen- 
erally known as psychical research dates from about 1880. In so far as 
that dealt with the problem of the occurrence of ESP, a wide range of 
testing materials was tried throughout the years. Many tests were 
made with numbers, playing cards, and other restricted items which 
gave results subject to a strictly quantitative evaluation. At the same 
time, many experiments based upon an indefinite class of items gave 
results of a qualitative sort: i.e., results which were not to be judged 
directly by mathematical criteria of significance. 

Early in the Duke investigations, the simple material now so widely 
used was adopted as the basis of these tests: five symbols upon cards 
which were usually used in packs of twenty-five, five of each symbol. 
Card material offered the most convenient means of obtaining a random 
series of events, essential to these experiments, and was at the same 
time readily adaptable to testing different aspects of the ESP hypothesis. 

In tracing the development of experimental methods, it will be con- 
venient to treat separately the advances affecting the relation between 
the subject and the card to be perceived on the one hand and, on the 
other, those concerned with recording and reporting the data. 


RELATION BETWEEN THE SUBJECT AND THE Event To BE PERCEIVED 


This discussion is confined entirely to methods for investigating 
clairvoyance, in which the subject was usually tested for his knowl- 
edge of symbols upon the concealed faces of shuffled cards. At the 
outset of the Duke work, the pack of heavy, hand-stamped cards was 
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placed face down before the subject. He called the top card, then re- 
moved it to one side without looking at the face, called the next before 
removing it, and so on through the deck. This was the “before touching” 
or BT condition. A natural development from this was to have the sub- 
ject call down through the shuffled pack without removing any cards, 
the DT condition. Corresponding to the BT procedure, open matching 
(OM) was introduced to permit a motor form of response making use 
of the card sorting principle. In this test, of course, the cards were 
sorted face down, usually at the subject’s own preferred speed. These 
tests were all used in an exploratory stage of the investigation. The 
results obtained from them were valuable chiefly in offering encourage- 
ment for further testing under more rigorous conditions. 


The first advances were in the direction of introducing distance be- 
tween the subject and the cards. Separations of one and two rooms, 
hundreds of yards, and even many miles were introduced without lower- 
ing the level of scoring of the successful percipients. 

The expediency of working in the same room with the subject led 
naturally to the use of opaque screens between the percipient and the 
cards. In early tests under this condition the subject handled his own 
cards under a screen which completely blocked his view of the process. 
Later, the cards were handled entirely by the experimenter, who held 
them out of sight behind a convenient screen. In the same class, ex- 
perimentally, as the tests with screens are those made with cards sealed 
in opaque envelopes. All these later conditions, providing complete safe- 
guards against the effect of sensory cues on the scores, have given re- 
sults which far exceed chance expectation. 


DEVELOPMENTS IN METHODS OF RECORDING 


At the beginning of the ESP investigations, the method of check- 
ing the score was that which generally prevails in other fields of sci- 
entific research; the investigator made his observations, checking the 
order of the pack of cards against the subject’s calls, with the latter per- 
son as an additional observer. The position of the correct calls in each 
pack was indicated on the record and the score for each run of twenty- 
five cards entered. More than ordinary care was taken to preserve all 
records intact and to report all without selection. 

The first advance to be made was the introduction of a third per- 
son as a witness to the testing and checking. This offered some addi- 
tional safeguards against the occurrence of incidental errors in record- 
ing. 








e- 
re 


b- 





Symposium oN ESP MetuHops 97 


An important advance was introduced naturally in connection with 
some of the experiments involving distance. This consisted of having - 
the records of the subject’s calls and of the cards made entirely inde- 
pendently. These records were made by different people (subject and 
experimenter) on separate record sheets with no possible influence 
of one record upon the other. 

Some progress has been made in the direction of entirely removing 
the human element from the recording process by the use of mechanical 
devices. Tyrrell in England has made considerable use of a machine 
which automatically selects an event to be perceived and records the sub- 
ject’s response. Several similar devices are in an experimental stage 
of development in this country and it is possible that something in this 
direction will constitute the final advance in recording safeguards. 


THE FurRTHER ADVANCE IN METHODS REPRESENTED BY THE 
INVESTIGATION HERE REPORTED 


The experiment to be described is one now in progress in the Para- 
psychology Laboratory as a joint project of Mr. J. L. Woodruff and 
myself. The work is a general investigation into the effect of size of 
symbols upon ESP scoring. However, this aspect of the work is not 
yet completed. The general method of testing ESP will alone concern 
us here. The plan of the remainder of this paper will be, first, to de- 
scribe the procedure in general terms; then to give the general results ; 
and, finally, to demonstrate concretely the actual procedure. 

Safeguards Against Sensory Cues. The experimental procedure 
used was a variation of that known as “screened touch matching” or 
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STM. A vertical screen (represented here by figure), two feet wide 
and 18 inches high is placed between the experimenter who deals the 
cards and the subject. This has a small aperture at the bottom allow- 
ing the experimenter to see the subject’s responses but blocking the sub- 
ject’s vision of anything on the experimenter’s side. An advance over 
previous uses of this condition consists in having the key cards by 
which the subject makes his indications hanging on the surface of the 
screen on little pegs out of sight of the experimenter handling the 
cards and in an order unknown to him. The key cards are rearranged 
just before the beginning of each run either by the subject or by the 
second experimenter, and the subject’s responses are made by pointing 
to positions corresponding to the key cards in the opening under the 
screen. The experimenter behind the screen (Woodruff) thus deals 
off the cards, following the subject’s pointer, into five piles without 
any knowledge of how the subject is identifying each card. This fea- 
ture avoids all possibility of such volitional errors on the part of the 
experimenter as might cause him to vary from the subject’s responses 
in a way to increase the number of hits. 

The cards to be guessed are shuffled five times by the second experi- 
menter (Pratt) and are given a final cut with a paper knife after the 
screen is placed in position for the run. The subject makes his responses 
at the speed he prefers. There are great individual differences but the 
average rate is around fifteen seconds for the run of twenty-five cards. 
The conditions, as will be clearly seen from the demonstration, guard 
against the use of cues on the part of either the subject or the experi- 
menters. 

Safeguards Against Recording Errors. These consist of features 
designed to eliminate any possibility of errors in checking arising from a 
knowledge of what would have been the correct distribution of cards, 
as well as features safeguarding against accidental or volitional selec- 
tion of the records in any manner. At the end of the run, Woodruff 
makes a complete record of the distribution of cards in the five posi- 
tions while the screen is still in place. Meanwhile, Pratt, who has been 
sitting on the subject’s side of the screen out of sight of Woodruff’s 
dealing of the cards, makes a record of the order of key cards for the 
run. These records are made on sheets which are officially stamped and 
marked with identical serial numbers. A different number is associated 
with each run by assigning in advance the length of series to be made 
in the experiment and obtaining numbered sheets from the laboratory 
recorder for the specific purpose of the predetermined series. These 
sheets are filed with the recorder when the series is completed and the 
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unbroken sequence of numbers is a completely reassuring safeguard on 
the question of selection of data. 

In addition to the record of the order of key cards, Pratt’s record 
shows on each sheet the name of the subject, the date, the type of test, 
and the initials of the two investigators. When each experimenter has 
completed his independent record, the two sheets with identical num- 
bers are clipped together and deposited in a locked box to be compared 
later and scored by a third person. 

The two experimenters then jointly check the run score by com- 
paring the distribution of the twenty-five cards with the order of key 
cards and sorting out and counting the hits. Each experimenter enters 
the score from this immediate count in a personal record book. 

When the record sheets themselves have been checked, a compari- 
son is made between the scores found from them and those kept by 
the experimenters at the time of the test. In case a discrepancy is found, 
the written records are consulted at once to see whether an outright 
oversight occurred in checking. If a re-count shows that such was the 
case, the score on the record sheet is revised. If the re-count does not 
show a checking error, the lower score is always taken as the official one. 
If, for example, the experimenters have a score of 6 for a particular run 
and the record sheets for that run give only 5, the investigators change 
their score to 5 and use that figure in all calculations from the data. Ac- 
tually, in 2,400 runs, only three such discrepancies, each of one hit, were 
found. The effect of these errors, if undetected, would have been to 
raise the score for the entire series by 3 hits. 

In every instance it has been evident that the error was made by 
one of the experimenters in making his independent record. The errors 
in recording were such as could be detected by the presence of six of 
one symbol and four of another in the card distribution, or by having 
one symbol appear twice in the record of the key cards. These facts 
show that, as far as our experiment goes, counting the score from the 
cards at the time of the experiment is more accurate than getting it 
from independent records. However, even in the face of the evident 
errors in the written record, the lower score was taken as the “penalty” 
for the discrepancy in all three cases. 

Results. With the procedure described, thirty-two subjects, chiefly 
undergraduate women students of Duke University, were tested be- 
tween the first of October, 1938, and the last of February, 1939. During 
this time, they made 2,400 runs or 60,000 individual trials. The hits 
numbered 12,489 or 489 in excess of mean chance expectation. If we 
evaluate this deviation by means of the formula for simple sampling, 
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S.D. = \/npq, we find that the S.D. is 98 and the deviation ratio, 5. 
However, as 2,400 runs was not set at the outset as the limit of the ex- 
periment, it is necessary to revise this deviation ratio to make it com- 
parable to the scale for simple sampling on which a value of 2.5 or 3.0 
is considered significant. 

Actually, the experiment as reported is made up of six sub-series, 
the length of each of which was fixed at the start. The problem of al- 
lowing for the effect of optional stopping after an indefinite number 
of such fixed sub-series is a new one for probability mathematicians. 
Dr. Greenwood of Duke University, on the basis of an unpublished 
solution of the problem, kindly calculated the probability for our series 
as corrected for optional stopping and found the value to be 5 x 10% 
instead of the uncorrected probability, 3 x 10-*. In other words, the 
uncorrected deviation ratio of 5.0 is equivalent to a deviation ratio for 
simple sampling of 4.4. Thus, the results are clearly beyond what might 
reasonably be expected by chance occurrence. In view of the conditions 
under which these results were obtained, we are compelled to regard 
them as “inexplicable” on any hypothesis unless it be that of an extra- 
sensory mode of perception. 


Demonstration.* 


SOME UNSOLVED PROBLEMS OF METHODOLOGY** 
C. E, Stuart 


The problems of extra-sensory perception arise from the daily-life 
events generally classified as “psychic experiences’ which can better 
be designated as parapsychic until their relation to the recognizedly 
psychic is cleared up. Examples of them are too well known to warrant 
discussion here. Most of them concern acquisitions of knowledge the 
means of which the subject cannot fit into his conception of the more or 
less orderly nature of his world. The psychologist, when consulted for 
an explanation, has the task of fitting the experience into his more ex- 
tensive knowledge of the orderly working of human nature. When his 
explanation is questioned, the only resort is to experiment. 

Since the experimentation is to discover whether there is some per- 
ceptual function generally common to the parapsychic experiences and 
not adequately deducible from contemporary systematic psychologies, 

* In the demonstration Mr. Burke M. Smith acted as subject, with Mr. J. L. 
Woodruff and Dr. Pratt as experimenters. The scores for the two runs were 


and 11 hits per 25. 
** Read by Dr. Bernard F. Riess. 
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the first problems of experimental method naturally follow the direc- 
tions outlined by Dr. Pratt in his paper. That is, can we, through lab- 
oratory experimentation, find some measurable performance which does 
not permit of reasonable alternative explanation and which might be a 
common factor in parapsychic experiences ? 

Granting the success of this endeavor experimentation might then 
proceed in either of two directions. First, we may accept what has been 
found in the laboratory test as operationally defined by the methods 
used, and try to find the further orderly relations concerning that prin- 
ciple. This is in effect what has occurred. ESP is what experimentation 
in extra-sensory perception under certain crucial conditions has pro- 
duced. What is the effect of distance on ESP? Of rate of performance? 
Of drugs? Etc. These findings become entirely empirical additions to our 
operational definition. 

The second direction of research might be to change the essential 
features of the ESP test to approximate more and more the daily- 
life situations from which the problem originated. The question would 
be whether, after all, this laboratory finding is related to the every- 
day parapsychic experience. In this case, our operational definition 
of ESP would be such common factors as could be pointed out in 
veridical parapsychic experiences. The results of laboratory tests would 
then be viewed in the light of their being more or less close approxima- 
tions to these “pure cases.” 

An example may make this thesis clearer. One of the outstanding 
features of the parapsychic experience is its generally sporadic nature. 
It just happens, apparently without much relation to the activity of 
the subject. The subject of an ESP test, however, reports at a certain 
time for a specified period of work. This latter condition is comparable 
to asking subject Jones to come up to the laboratory at 2:30 Wednes- 
day afternoon and have a veridical parapsychic experience while two 
observers watch him and check up. 

Now if the sporadic, inconstant nature of the parapsychic experi- 
ence is a real quality of the process of extra-sensory perception, then 
the laboratory appointment is methodologically bad, and some technique 
should be devised to allow for that quality in the working principles of 
the experiment. 

The present discussion is concerned then with noting both discrep- 
ancies and similarities between the ESP test and the every-day para- 


psychic experience with a view to seeing what methodological advances 
in the laboratory tests are indicated. 
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1. The parapsychic experience is sporadic, inconstant, and generally 
unreliable as a guide to action. This is true even of trance mediums 
and professional clairvoyants. Although the difficulty of objective tests 
of professional clairvoyants is notorious, even when their performances 
are given every possible credence, many of their assertions are simply 
untrue. In general, however, we may note that some individuals appear 
to have more such experiences than others, and some seem to have 
more reliable experiences than others. 

An ESP test, then, should show wide differences in individual per- 
formance. And they have done so. The methodological question is that 
of finding an objective technique for selecting individuals of the greatest 
reliability. The tests have also shown that individuals are inconsistent 
from day to day, and that even reasonably consistent high scores de- 
cline or disappear with time and monotony. Techniques for testing the 
predictability and discovering the causes of such scoring changes and 
weighting the results accordingly are needed. These techniques might 
involve experimental criteria (such as performance in some non-test 
task) or delicate mathematical methods for analysis of scoring changes. 

2. The parapsychic experience usually deals with events of per- 
sonal emotional significance to the subject. The ESP test, with its 
simple guessings of uncomplex symbols, varies widely from this con- 
dition. And one could question why the test situation works at all. But 
frequently events of emotional significance are objectively trivial. What 
has been noticed is the importance of the influence of the experimenter 
and the social situation upon the subject’s performance. Differences in 
the nature of the stimulus objects as such seem to make no reliable dif- 
ference in the scoring. 

What is suggested is that the subject should be personally interested 
in the outcome of the tests and should feel that the experimenter is 
himself interested similarly. The problem here is the objectification of 
these vague social skills into some interest-giving element in the situa- 
tion itself; that is, to find the psychological essentials for such mutual 
interest and to objectify them in the experimental methods themselves. 

The importance of the emotional background of the laboratory sit- 
uation is easily deducible from wholly psychological considerations. If 
we forget the problem of ESP for a moment, Dr. Pratt’s demonstration 
could be described as simply the presentation to a subject of an impos- 
sible task. Dembo used an essentially similar technique in Lewin’s lab- 
oratory for the study of the genesis of anger. The only difference be- 
tween the ESP test and a pure frustration situation lies, therefore, in 
the role played by the experimenter. 
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The question of using material of directly emotional significance to 
the subject remains, however, largely unexplored, so far as quantitative 
estimation with mere symbol tests are concerned. One utilization of this 
factor probably occurs in the attempted reproduction of free drawings, 
wherein the emotional significance of the stimulus object can at least 
be postulated. Conditions for control of alternative hypotheses and for 
accurate statistical evaluation have recently become available, and wider 
use of the method is to be expected. What is still needed is a valid 
method of psychological analysis. The interpretation of drawings, much 
as the latent content of a dream is inferred from its manifest content, 
might seem to match the conditions of the parapsychic experience more 
closely than any other evaluation. Further consideration of the generally 
emotional aspect of the test situation might involve adaptation of word 
and free association techniques. 

Specific emotional factors can and have been introduced such as re- 
wards, competition, “play,” and the calling upon the pride of the sub- 
ject or his friendship for the experimenter. These latter, however, vary 
in effect with the social skill of the experimenter. 

3. Another feature of the parapsychic experience is its occurrence 
frequently during periods of general abstraction. The ESP test situa- 
tion may vary widely from this requirement, but it has been generally 
noted that distractions decrease scoring. While the experimenter may 
strive to keep the subject from feeling self-conscious, he must fre- 
quently fail. The need suggested is for techniques which allow the sub- 
ject to work alone, and at his (rather than the experimenter’s) con- 
venience. This need may probably best be met in one way by wholly 
mechanizing the test procedure. Several ESP machines have been de- 
vised, and the requirements such machines must meet have been re- 
cently summarized by Dr. Rhine. The machine should be technically 
capable of excluding alternative hypotheses adequately, should be com- 
pletely reliable in performance, and should be amenable to operation 
with little attention or physical effort on the part of the subject. But the 
use of this method is as yet untried and many others may be needed be- 
fore the problem is adequately solved. 

Many of the foregoing suggestions might well be incorporated in 
solution of a problem which must ultimately concern the ESP investiga- 
tion—a universally repeatable experiment certain to yield positive re- 
sults. There is no a priori reason to expect that such an experiment may 
be set up with the methods now available. And even the formulation 
of requirements would scarcely be agreed upon today. But we do know 
that the repeatable experiment must meet rigorous requirements for the 
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exclusion of all but the ESP hypothesis yet lay as few restrictions as 
possible upon the hypothetical possibilities of the nature of the ESP 
process itself. 


THE ROLE OF SELECTION IN ESP DATA* 


VERNON W. LEMMON 


Criticisms of ESP experiments by psychologists have fallen into 
three major categories: a) mathematical evaluations of departures from 
chance expectations; b) the possible effects of sensory cues; 3) the 
effects of various kinds of non-random selection of data. 

Since mathematicians have recently shown considerable interest in 
the first category, it is perhaps time for psychologists gracefully to re- 
tire, and leave this field to those more competent to handle it. The ques- 
tion of sensory cues has been discussed at great length, and researchers 
in ESP have shown a willingness to modify their techniques in order 
to avoid such effects. 

The possible effects of selection were discussed by the present writer 
in a previous article, and the question has been persistently revived by 
many other critics. At the same time the ESP researchers have con- 
sistently denied that these criticisms have any validity. It is quite pos- 
sible, therefore, that the critics and the criticized have not fully under- 
stood each other, and that the matter deserves further clarification. 

Clarence Leuba, in a recent article,? states that non-chance results 
can be obtained from chance data if a) “lucky’’ subjects are used, and 
if b) runs are stopped at favorable points. These two conditions are 
probably not entirely independent. If chance alone is operating, there 
can be no truly “lucky” subjects. To be sure, a subject may have a long 
run above chance, but if he continues long enough, there will certainly 
be compensating runs below chance. A subject can only be made to ap- 
pear lucky by arbitrarily stopping his run while he is still scoring above 
chance. Thus the first part of Leuba’s criticism seems to be included in 
the second. 

The ESP researchers have steadfastly maintained that a true chance 
series could be stopped at any point, and resumed at any point, either 
with the same or a different subject, without changing its chance char- 
acter, since the earlier runs have no influence whatever upon the later. 


* Read by Mr. Burke M. Smith, Duke University. 

1 LEMMON, VERNON W. Extra-sensory perception. J. Psychol., 1937, 4, 227-238. 

? LEUBA, CLARENCE. Has recent research undermined the evidence for extra- 
sensory perception? J. Appl. Psychol., Vol. XXII, 6, Dec., 1938, 549-553. 
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These contentions are of course true, but they do not cover fully all 
aspects of the question. 

Let us assume that a subject is making pure chance guesses, and that 
a cumulative average of his runs is being kept. This average will fluctu- 
ate both above and below chance expectation. Let the run be termi- 
nated while the cumulative average is above chance, and the value en- 
tered in the data book. Now let a new run be started, with either the 
same or a new subject. These guesses will be totally independent of the 
previous guesses ; the cumulative average may at first trend high, low, or 
normal. However, if the series is only continued long enough, there 
will certainly come a time when the cumulative average will experience 
a fluctuation above chance expectation. Let the run be terminated at 
this point, and the value of the average entered in the data book. It is 
easy to see that if this procedure should be exactly repeated a hundred, 
or a thousand, times, the data book will still contain nothing but “plus” 
values. At the same time the averages, taken as a whole, will include 
every guess that has been made, and it can truthfully be claimed that 
not a single case has been neglected. 

It has repeatedly been claimed that if every case is included in the 
results, high and low averages will tend to occur in equal numbers (if 
chance alone is operating). Yet the discussion above shows that this is 
not necessarily true. Where is the paradox? 

It seems to the writer that the data book averages just described 
really constitute a series of “readings” or “observations” on a fluctuat- 
ing quantity, and that, furthermore, they constitute a selected series, 
even though every case has been included. It would have been equally 
possible, with the same subjects, to have selected a series of observa- 
tions every one of which would be below chance expectation, or exactly 
at chance expectation, and in these cases also, every guess would have 
been included. 

Selection is thus not avoided by merely including every guess which 
has been made. It is necessary to go a step further, and make sure 
that the “observation points” (i.e., the stopping points of the runs) 
are not selected either. This can be done by stopping the runs according 
to a criterion which is totally unrelated, directly or indirectly, to the 
state of the score. Practically, the easiest way to accomplish this would 
seem to be to determine in advance of each test just how many guesses 
are to be included, as has been suggested by practically every critic 
writing on this topic. Even this method is not entirely free from possible 
criticism. For example, the experimenter might decide that since the 
subject was doing very well today he would set a limit of twice as 
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many runs as usual. Here selection has crept in again. Perhaps a ma- 
chine which would ring a bell after a variable interval, unknown to the 
experimenter, would be the only really satisfactory solution. 


DISCUSSION 
The Chairman: 


Because Dr. Lemmon’s criticism raises mathematical questions, I will ask 
Dr. Greenwood or Dr. Greville to open the discussion. 


Dr. Greville : 


I think there is a fallacy in the argument presented in Dr. Lemmon’s 
paper. It is a rather subtle one, but I think it is possible to make it clear. 
Suppose for the sake of argument that we are going to make a certain num- 
ber of guesses; at least we are going to terminate this series of guesses at 
a point where the average score is above the chance level. We are going 
to get a certain average score and then we are going to start either immedi- 
ately or at some later time another set of guesses, keeping this up until we 
reach a point where the average is again above chance level. We are going 
to put that down, as he says, in the data book in a similar way. We are thus 
going to get a lot of other score averages all of which will be above chance. 
We will put all of those down. When we get through, we will average all 
of them up and get a result which will certainly be above chance. Now that 
would certainly follow, if the unbroken series of averages above chance is 
given. Let us examine this assumption. If nothing more than chance is op- 
erating we might properly stop for a while for any reason and then go on 
again. We might change to another subject; (that is the same thing as if 
we used a different pair of dice, if we were throwing dice; or if we were 
in a bridge game and someone got up and walked around the table—some 
people think that will change your luck). But for whatever reason we stop, 
suppose we think of: all of these trials as one long series that we wanted 
to divide up into parts; and suppose we are free to choose our points of divi- 
sion. Now let us say the average of this whole series is slightly above chance, 
but if we compute the average at various points, as long as it fluctuates 
sometimes above chance, of course we can pick out places where it is above 
chance. In that way we would have a series of blocks of scores in the data 
book that would add up to something above chance. On the other hand, we 
could just as well divide the total series at points that would average below 
chance, picking out the bottom of these troughs (referring to diagram). 
(Of course the last one would have to be above.) In this case all of our av- 
erages in the data book, except the last one, would be below chance—only 
the last figure being above—yet we get the same total as we would by the 
other procedure. I don’t think that criticism is really mathematically sound. 


Dr. Rhine: 
Have you anything to add, Dr. Greenwood ? 
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Dr. Greenwood: 


Nothing except possibly to make the argument more devastating. And 
that is because it is not certain that in a finite series such characteristic 
division points as Lemmon requires could be found. But granting that fact, 
all that Dr. Greville has just said would still be good. 


Dr. Rhine: 


One thing you would grant, I suppose, is that at the end of the series 
one can stop, if he is allowed optional stopping, with a little deviation above 
or a little below, as may be desired. This point you have already discussed 
and some of the advances in mathematical methods you have presented today 
would be applicable as a slight correction. 

The meeting is now open for general discussion. If you will be good 
enough to give your name and institution for the record when you speak, 
I shall appreciate it. 


Dr. Riess (Hunter College) : 


I am not, clear on Dr. Pratt’s new method as to how he is taking care 
of the criticisms we have been hearing concerning possible auditory cues 
from the shuffling of cards. How does he guard against the auditory cues 
from the adhesion of cards of different symbols? That was a slight criticism 
made at Columbus—that one of the hypotheses for high scores in ESP is 
the possibility of such cues in card shuffling and card drawing. In the 
method that Dr. Pratt is using, it seems to me the possibility is there for 
that type of objection—not that I think the criticism iself is worth very 
much—but it has been made as one which we do have to answer. 


Dr. Pratt (Duke University) : 


Mr. Chairman, if Dr. Riess could make more explicit the principle sup- 
posedly involved, I think we might see more clearly just what this criticism 
might be worth. As far as auditory cues during the shuffling of the cards 
which the subject is guessing are concerned, that is done by the second ex- 
perimenter and is not of importance for determining the order in which 
the cards are at the time the run starts, since, as we know, they are 
cut with a paper knife behind the screen as the last thing before the experi- 
ment begins. How there could be any indication from the sound of a paper 
knife in cutting the pack as to what the card on the top or the bottom of 
the pack is, I don’t know; but perhaps it is possible to conceive that the paper 
knife vibrates differently for each symbol. At some point in these criticisms 
judgment has to come in. 

Now, it is perfectly possible to devise an experiment about which no one 
would ask this question at all, because it would be obvious that the distance 
was such as to make it absolutely ridiculous. We have used the present pro- 
cedure for the advantages, we might say, which it offers the subject and 
for the difficulties or the attenuation which it gives to the criticisms. In 
our procedure it is easy to see that a very complicated interchange of cues 
would be necessary if the results were to be affected at all. We would have 
to imagine, if any assistance is to be obtained from cues, that the subject 
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first gives the experimenter some hint of the order of key cards. The ex- 
perimenter would have to use the information so gained in misplacing, 
consciously or unconsciously, the cards from the packs or in indicating by 
cues to the subject what the next card really is, the subject then making his 
next indication so as to get a hit. All this would have to occur in some- 
thing like half a second. It becomes for this procedure simply a question 
of judgment as to whether such cues could occur and affect the results. 

Dr. Riess: 

I don’t want to be a Devil’s advocate but these possibilities come up 
all of the time in criticism. I am not sure whether Woodruff was able to see 
you as you sat in the room back of the subject but that is a criticism that 


is bound to come up when people who are more than unusually skeptical 


of the experimenter’s honesty are involved and there are criticisms of that 
kind. 


Dr. Pratt: 
Yes.... 


Dr. Riess: 


I would like to put in a word of warning against the use of any method 
which is open to criticism of that type. 


Dr. Pratt: 


Well, the word is well received as far as I am concerned. At present 
our experiment is in a stage in which perhaps no such criticism would be 
raised, but we are just starting. The results haven’t yet reached the point 
that we may consider them, I will say. 


Dr. Miner (University of Kentucky) : 

May I simply make a suggestion as to what I think would be interest- 
ing to those who are studying ESP. I assume in some series or trials or 
runs you occasionally have a failure. Is that possible? 

Dr. Rhine: 

Yes. 


Dr. Miner: 


My suggestion is that, when you do have a failure, you publish a list of 
the explanations of that failure which you might give—just a list of the ex- 
planations which you regard as sound for that failure. 


Dr. Rhine: 


May I ask one question for further elucidation? Do you mean the ex- 
planations which we feel certain are experimentally established or such 
hunches, as we might have? 


Dr. Miner: 


Your various hunches as to the explanation for that failure, just a list 
of what you might regard as explanation for that failure. 
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Dr. Rhine: 


That might lead to the charge that such explanations are “alibis.” Has 
anyone any comment to make on that? 


Dr. Greville : 


I suppose this might be done in order that other people trying to repeat 
the experiment can avoid those same causes of failure? 


Dr. Rhine: 


Would that not be a part of the need for a general discussion of all that 
we know, suspect, and guess as to what contributes to either failures and 
successes ? 


Dr. Miner: 


Your suggested explanation of failures and successes. I think it is de- 
sirable that we have some list of interpretations which might be used in 
judging ESP. 


Dr. Rhine: 


I might say that we contemplated a survey of research articles beginning 
back with the impressions of those that worked with the subjects long be- 
fore we began at Duke, adding on such tentative judgments as more recent 
work has led us to have. I think we have been a little shy about presenting 
this for the reason that it exposes us a good deal to the suggestion that 
we are trying to apologize for failures or trying to read in unestablished 
conclusions. But I think the project you mention is a very important one. 


Dr. Miner: 


I don’t object to delay until you have a critical series of failures as well 
as a critical series of successes. 


Mr. Crissy (University of North Carolina) : 


There is one point I would like to raise. I don’t know enough about 
your experimental set-up to be sure whether it is a contribution or not. I 
am interested in tests and measurements; hence I can’t help thinking of the 
present problems in terms of, say, a testing procedure. Let us say that 
someone shuffles and records the pattern in ten decks of cards and that the 
record of this pattern is duly signed and sealed. A week later in the lab- 
oratory an experimenter uses these set cards as the run cards and has a 
subject call these cards by one of the acceptable techniques. After collecting 
the data (records of the 250 calls) an inter-item matrix could be easily com- 
puted (250 x 250) and analyzed for internal consistency by the Kuder- 
Richardson procedure (Kuder, G. F. and Richardson, M. W.: The Theory 
of the Estimation of Test Reliability: Psychometrika: vol. 2, No. 3, Septem- 
ber 1937). This treatment, I believe, would place emphasis upon the indi- 
vidual call as the elements of ESP (rather than the run as was reported 
by the gentleman from Columbia who used odd-even runs). If you used 
such a procedure on the data from Dr. Pratt’s experiment I believe you 
would get an estimate of reliability of a test which you would designate 
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probably a test of ESP. I don’t know whether this would be contrary to 
the usual experimental set-up. I must confess, however, that when I read 
some of the articles | kept wondering to myself: How reliable is the test? 


Dr. Miner: 


In view of the large amount of work you are doing, there are some 
things which are of very definite advantage that would come out of it, if 
you are willing to devote the time. I think the main difficulty of psycholo- 
gists is wondering whether they should devote that much time. Two things, 
it seems to me, might come out of the research: First, the stimulation to 
mathematical analysis and secondly some idea as to the nature of the pat- 
terns of guesses if these were treated really as patterns of guesses. 


Dr. Rhine: 


Is there any comment from anyone working in the field? It seems to 
me there might be something said first, as to the question raised by Dr. 
Crissy. How close are you coming to meeting this question in your A and B 
series, Dr. Pratt? 


Dr. Pratt: 


I am afraid it doesn’t come very close. 


Dr. Rhine: 


Perhaps you will explain what your procedure is. 


Dr. Pratt: 


For a period in our work we divided our subjects on the basis of per- 
formance up to that time into A and B classifications, A being those that 
we thought would do better in the remainder of the tests, B, those that we 
suspected would do poorer. For a time we found a slight difference in 
favor of the A’s in subsequent performance, but not a significant difference 
between the two groups. Our subjects, in other words, were selected with- 
out reference to ESP ability, and were largely unselected throughout the 
period of working. It seems that, whatever deviations we obtained and 
however they are to be explained, there was no reliable basis of prediction 
on past results as to how the subjects’ scores would be divided in the future. 
However, an acceptable measure of reliability would depend upon frac- 
tionating the data within the same day, I judge, or in such a manner that 
you could be sure the conditions were more nearly comparable. It wouldn't 
be desirable to try to determine the validity of the test by taking the per- 
formance of a subject for one month and comparing it with the perform- 
ance for the next month. We could make some such division within the 
day, taking even and odd runs for the same subject, and arrive at a con- 
clusion as to whether there is any significant correlation. 


Dr. Riess: 


I have been trying to get at some measure of reliability and the correla- 
tion for one subject who got a significant score in my series on odd versus 
even trials for the calls alone, uncorrected, was .71. Of course that is an 
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exceptional case. I am going to do the same kind of split-half technique for 
the subjects who score chance. 


Mr. Crissy: 


One of the things of course I had in mind there was this interesting 
sidelight, a computation 4 la Kuder-Richardson could then be compared 
with reliability computed for the same data but by some fractionation of 
runs, say odds against evens. Also you would find it appropriate probably 
to apply both techniques to “good” runs (demonstrative of ESP) and to 
“bad” runs (chance figures) and see whether the “test” measures ESP as 
consistently as it does the lack of ESP. 


Dr. Miner: 


One of the suggestions that came out today should be published pretty 
broadly, and that is the suggestion that the phenomenon, so far as you can 
judge, is sporadic and therefore up to the present time you can see no use 
for it except from a pure science point of view. It seems to me that you 
might caution the public from trying to put ESP to use before it were 
justified. 


Dr. Rhine: 


We waste a lot of postage and paper trying to discourage some en- 
thusiasts on that point. A general warning should be very helpful. 


Dr. Shea (U. S. Forestry Service) : 


Dr. Rhine, have you ever noted in your experiments degrees of what we 
call success and failure of people when they are ill for any reason? I think 
that might throw some light on this problem. That is, take any of the minor 
illnesses—headache, menstrual period, or anything else—and observe devia- 
tions that occur in the general trend of successes or failures during periods 
of so-called illnesses. 


Dr. Rhine: 


There have been a number of observations made on the health of the 
subject, incidentally in the course of a long series of tests. There have 
never, of course, been any experimental illnesses induced. One is likely to 
run into the objection that the experimenter, knowing what the score is 
and observing the illness, unless it is carefully observed and recorded before 
any tests are made on a given day, may give an interpretation that is not 
entirely objective. In the earlier investigations which were not done under 
conditions which we now impose on ourselves but which we feel are very 
much worth considering in view of the results from the later work done 
under better conditions, there were several subjects ill at one time or an- 
other. The most common illnesses were colds and influenza, and there was 
notable falling off with such indispositions. That was published for what 
it was worth, not as a conclusion of the final order that we would like 
eventually to reach. 

The other point I would like to mention is the case of two subjects who 
have been studied at considerable length and reported to have lost their 
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ability to perform following medical treatment. In these instances the ill- 
ness was apparently a favorable condition. That is naturally a generalization 
which we would not want to make for other cases. In the work reported 
by Dr. Riess he got remarkable results under excellent conditions until his 
subject was taken to a physician and treated. (I believe the diagnosis was 
hyperthyroidism). Results of tests made following treatment approximated 
chance. They had been exceedingly high before. There is another case in- 
volving thyroid disorder which Dr. Drake studied. His investigations showed 
apparent ESP but he reports in a later unpublished work that his subject, 
a boy, having undergone treatment for hypothyroidism is now unable to 
do the formerly good scoring he had done. 


Dr. Greenwood: 


Practically all of the experiments have been carried out with the view 
to getting a significant deviation ratio on the mean. I wish to suggest that 
there may be other measures of ESP performance, perhaps unknown at 
present, due to some unknown characteristics of ESP itself. In fact, there 
might be a dozen entirely different statistical measures of such a faculty. 
For example, the variance would measure an unusual dispersion of the 
scores. New hypotheses relevant to ESP can call for new measures of it 
and vice versa. Theoretical considerations and generalizations with no 
more apparent application than a game of chess have frequently pointed the 
way to correct hypotheses. 


Dr. Rhine: 


That has a good echo in my mind from the views of a group of psychol- 
ogists at another university. I don’t know that I should speak of their work 
before publication; but they report a very subtle effect that is not dependent 
for evaluation upon the methods we ordinarily use, the deviation from the 
mean. One Of these experimenters is present. I wonder if you would want to 
say anything about your findings, Mr. Taves? 


For Mr. Taves’ approach the reader is referred to the article on p. 38 ff. 
of this issue. 


Dr. Rhine: 


If it isn’t clear, that article is coming out in the JouRNAL oF PaRapsy- 
CHOLOGY in the next number. It can be read in full detail. I take it this is 
such a case as you had in mind a moment ago, Dr. Greenwood. 


Dr. Greenwood: 
Yes. 


Dr. Rhine: 


I think we need to keep in mind, too, such possibilities as this: Two 
persons in a delicate performance have very different favoring conditions. 
For example, some people think best, they say, with the radio turned on; 
some people think best with it off. (You can think of better illustrations 
than that.) Suppose our actual experimental conditions happened to divide 
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subjects in the respect of favoring and hindering them, and we pooled the 
group of data for evaluation. It would not be striking since some of the 
subjects might have had negative deviations by reason of these conditions 
and the two tendencies cancel each other. Or take a single individual; on 
some days things will make him favorably inclined toward good perform- 
ance—events of the day, the atmosphere of the laboratory, and so on. The 
conditions of the next day may be the reverse, and scores be affected ac- 
cordingly. Or changes may occur in the course of the sitting itself within 
the hour and conceivably even in the run of 25 calls. We need to find ways 
of measuring the data delicately enough to catch any such effects. 


Dr. Greville: 


Dr. Rhine, there is one other point at the end of Dr. Lemmon’s paper. 
He speaks about the importance of avoiding selection and he says the easiest 
way practically to accomplish this would seem to be to determine in ad- 
vance of each experiment just how many guesses are to be included (as 
nearly every critic on this topic has suggested.) He says further that even 
this method is not entirely free from error. For example, the experimenter 
may decide to stop at advantageous points in the day’s series, thus ending 
each day with a favorable balance, yet still keeping within the set total for 
the series. 

Since Dr. Lemmon is not here today, perhaps it is unfair to go into this. 
Doubtless what he means is that we might make the total deviation of the 
entire series greater or less by merely lengthening or shortening the sec- 
tions in which this series is carried out. The fallacy of this was pointed 
out earlier this afternoon. There is no basis for supposing that because a 
subject has been doing well' by chance alone that he will do as well on the 
next run made immediately instead of holding over to a later period. This is 
an old idea that often has been expressed (but does not hold water), namely: 
If you toss a coin and get heads nine times, that makes it more likely you 
will get heads another time. But if nothing but chance is operating, we have 
no reason whatever to suppose that because the first fifty runs were good, 
the next fifty are going to be good, just because he is going to do the next 
fifty right now instead of waiting until tomorrow. The only hypothesis on 
which we could make such an assumption would be that the subject has 
the same capacity for making the second run good that he had for the first 
one. There is nothing in chance in the shuffle of a deck of cards that is going 
to do it. 


Dr. Miner: 


If the subject has the idea whether he is going to be good or bad, fatigue 
or various factors make a difference, so he might tell before you even did 
the first series. 


Dr. Greville: 


If it is just chance operating, fatigue and those other factors would 
have no effect on that. 
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Dr. Calvin (Yale University) : 


I wanted to ask whether you are justified in assuming in a repeated 
series of this sort that chance alone might be operating or is that what you 
are trying to disprove? 


Dr. Greville : 


What ESP investigators are trying to show is that chance is not the 
only factor involved nor the chief factor involved. So we are making the 
hypothesis that chance is the chief factor and then show that that leads 
to inconsistencies; therefore it is necessary to postulate some other factor. 


Dr. Rhine: 


I wonder that no one has pitched into the final proposition made by Dr. 
Greville in his paper that we can practically disregard this section in here 
(indicating on diagram the remainder of the data of a series after the most 
favorable stopping point). What Dr. Greenwood and he are trying to get 
at is a safe limit which will enable us, let us say for any purpose of evalua- 
tion of the data, to move up the gauge of measurement with entire safety 
into whatever area general inspection would indicate might be of interest. 
You will notice what they have done is to raise the critical ratio required 
for significance; but the method may be quite useful even so. Dr. Green- 
wood’s measure is getting at another type of problem, the type of thing 
which Dr. Pratt and Mr. Woodruff were dealing with. This deals with the 
adding on of runs by the block setting up, say, 300 runs per block and after 
that adding another 300 as long as desired. Dr. Greenwood has an un- 
published method of making the correction necessary for optional stopping 
in blocks. 


Dr. Greenwood: 


I want to make it clear that the two methods measure similar situations. 
The difference is that Dr. Greville treats blocks of one run as contrasted 
with my blocks of several runs each. 


Dr. Rhine: 


Yes, but the problems are different for us. It is different on the one 
hand to be able to go back over a long series and pick out a section for 
separate evaluations and, on the other, to be able to add on blocks of data 
as long as we want to. 


Dr. Greenwood: 


Both have their disadvantages. You might, in my case, get a pretty good 
approximation by not having so many stopping points; and in Dr. Greville’s 
case, with not as close an approximation, he could have the advantage of 
many more stopping places. 


Dr. Greville : 


My method is perhaps a little more flexible from an experimental stand- 
point. It raises the critical ratio perhaps too much. I can’t say just what 
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the probability is. I can say it is less than a certain figure, and that figure 
may be higher than you like. Dr. Greenwood’s method is not as flexible in 
use, but it doesn’t raise the ante so much, you might say. For instance, if 
you use my method, this data that Pratt has put on the board, I couldn’t 
declare with confidence on that basis that it would be significant at all be- 
cause my method, instead of giving an accurate estimate of the probabilities, 
just says I know it is less than some figure which may be much too large 


a figure. 

Dr. Rhine: 

This measure here would probably be a significant one even by your 
criteria, would it not (5.0)? 

Dr. Greville: 

It probably would, but I can’t be sure of that. 


Dr. Rhine: 


You can see, I think, that those of us engaged in this research are eager 
to get all possible assistance from you and I don’t want to suggest dismissal 
as long as there is likely to be further constructive comment. (Pause) Thank 
you very much. 
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LETTERS AND NOTES 
To the Editors: 


The discovery of a new and astounding set of scientific facts is 
inevitably followed by a wave of overenthusiasm. Overenthusiasm re- 
sults in undesirable publicity, commercial exploitation, incompetent 
experimentation, and exaggerated claims. In general, scientists are 
men of high repute and character, but they are also human. Psychology 
has much to learn from the older science of medicine. The history of 
medicine is replete with sensational discoveries, universal panaceas and 
revolutionary cure-alls, most of which have been discredited after sub- 
sequent study demonstrated them to be ill-advised or fraudulent. The 
motivations for making premature or fraudulent claims are usually 
obvious. The discoverer gains transient attention, repute or material 
rewards. Medical men are no different from psychologists or para- 
psychologists. Psychology is too young a science to have gained bitter 
experience with fraudulent or ill-advised claims, but the numbers of such 
claims will undoubtedly increase if it becomes profitable to make them. 


Organized medicine has established an elaborate system for veri- 
fying the facts concerning any particular claim or set of data. Experi- 
ments are repeated in many different laboratories. Unusual clinical 
cases or specimens are exchanged between different hospitals or univer- 
sities. The American Medical Association has appointed special com- 
mittees to act as referees on certain types of claims. Unusual claims 
are not accepted unless accompanied by irrefutable proof, or unless they 
are subject to subsequent verification. 


Parapsychology is particularly vulnerable to suspicion and doubt 
from other professional groups. As uncontrolled experimental factors 
have been progressively analysed, the average number of hits in long 
series of runs has steadily diminished. Only a few isolated subjects 
have obtained extra-chance ESP scores in long runs. Unfortunately, 
these remarkable subjects have not been sent around to neutral lab- 
oratories for retesting. Nor have they been studied over a long enough 
time to determine the constancy of their abilities. There has been too 
great a tendency to dwell on one set of results, such as the Riess series, 
which cannot be verified by retesting. 
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It cannot be too strongly emphasized that a remarkable subject who 
consistently scores far above chance expectations is such a scientifically 
important and rare case as to warrant the most careful study by neutral 
referees. Future experimenters should be obligated immediately to an- 
nounce their discovery of a remarkable subject so that neutral referees 
could vouch for the validity of the claims. In the event that other re- 
markable subjects are discovered, it would be worthwhile for the 
American Psychological Association or other interested group to set 
aside funds thoroughly to investigate each experiment. These com- 
ments are not intended to cast suspicion on any set of experimenters or 
data. They merely attempt to state the most common criticism directed 
against ESP experiments by scientists in related professional groups. 

Finally, it would seem important to allow a gifted ESP subject the 
opportunity to make predictions which might result in his economic 
gain. It would indeed be remarkable if ability in extra-sensory per- 
ception is restricted to ESP card predictions. The ability might be 
applied to games of chance, or the prediction of economic trends, where 
material gain would result from extra-chance predictions. 

F. C. THorne, M.D., Ph.D. 


Professor Gardner Murphy June 3, 1939 


Department of Psychology 
Columbia University 
New York, N. Y. 


Dear Prof. Murphy: 

The members of our committee have criticized the Murphy and 
Taves manuscript, “Covariance Methods in the Comparison of Extra- 
Sensory Tasks.” Joint communications have been prepared by Drs. 
Hilgard and Kennedy, and by Drs. Dick, Long, and Sells. Individual 
reviews were submitted by Drs. Gibson, Lorge, Willoughby, and Volk- 
mann. 

Inasmuch as most of the specific points raised in relation to the 
original manuscript, such as ambiguities in the labelling of tables and 
charts, and in the identification of specific data, have been accepted and 
corrected prior to publication, we shall summarize in this letter only 
the more important methodological criticisms raised by various mem- 
bers of the committee : 

1. The description of experimental procedures appears sufficiently 
detailed and complete to permit duplication of the experiment by an- 
other laboratory. 
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2. No questions concerning faulty experimental technique have been 
raised by the committee. However, a number of points concerning 
treatment and interpretation of results have been made. 

3. The principal criticism concerning treatment of data was that 
adequate controls of possible linkage among target decks were not 
shown in the original manuscript. The members of the committee re- 
quested such controls in order to determine whether or not the reported 
covariance results for calls were spurious. Hilgard and Kennedy; 
Dick, Long, and Sells; Willoughby ; and Lorge requested specific con- 
trols of these factors.* 

4. The original manuscript based the proof of the covariance hy- 
pothesis upon the concomitant variation of playing-card (color) scores 
and circle-cross and circle-blank scores as dependent variables. Dick, 
Long, and Sells; Willoughby; and Lorge specifically suggested com- 
plete rotation of all variables, playing-cards (color and suit), ESP, 
circle-cross, and circle-blank as dependent and independent variables.** 

Saut B. SELLs. 

*The additional controls of linkage, requested in item No. 3, appear on 


pp. 62-65. nice 
** The intercorrelations requested in item No. 4 appear on page 57. 





